Technical  Report  1141 


Coding  Verbal  Interactions  In  A  Prototype  Future  Force 
Command  And  Control  Simulation 


Paula  J.  Durlach 

U.S.  Army  Research  Institute 

Laticia  D.  Bowens  and  John  L.  Neumann 

University  of  Central  Florida 
Consortium  Research  Fellows  Program 

Thomas  J.  Carnahan 
Western  Kentucky  University 
Consortium  Research  Fellows  Program 


January  2004 


20040315  011 


United  States  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences 


Approved  for  public  release;  distribution  is  unlimited 


U.S.  Army  Research  Institute 

for  the  Behavioral  and  Social  Sciences 

A  Directorate  of  the  U.S.  Army  Human  Resources  Command 


ZITA  M.  SIMUTIS 
Director 


Technical  Review  by 

William  R.  Sanders,  ARI 
Donald  R.  Lampton,  ARI 


NOTICES 

DISTRIBUTION:  Primary  distribution  of  this  Technical  Report  has  been  made  by  ARI. 
Please  address  correspondence  concerning  distribution  of  reports  to:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  Attn:  DAPE-ARI-PO, 

5001  Eisenhower  Ave.,  Alexandria,  VA  22304-4841 . 

FINAL  DISPOSITION:  This  Technical  Report  may  be  destroyed  when  it  is  no  longer 
needed.  Please  do  not  return  it  to  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences. 


NOTE:  The  findings  in  this  Technical  Report  are  not  to  be  construed  as  an  official 
Department  of  the  Army  position,  unless  so  designated  by  other  authorized  documents. 


REPORT  DOCUMENTATION  PAGE 


1.  REPORT  DATE  (dd-mm-yy) 
January  2004 


4.  TITLE  AND  SUBTITLE 


2.  REPORT  TYPE 
Final 


Coding  Verbal  Interactions  in  a  Prototype  Future  Force 
Command  And  Control  Simulation 


6.  AUTHOR(S) 

Paula  J.  Durlach.{U.S.  Army  Research  Institute);  Laticia  D. 
Bowens,  &  John  L.  Neumann  (University  of  Central  Florida);  & 

Thomas  J.  Carnahan  (Western  Kentucky  University). 


3.  DATES  COVERED  (from. . .  to) 
October  2001  to  August  2003 


5a.  CONTRACT  OR  GRANT  NUMBER 


5b.  PROGRAM  ELEMENT  NUMBER 
0602785A 


5c.  PROJECT  NUMBER 
A790 


5d.  TASK  NUMBER 
211 


5e.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
Simulator  Systems  Research  Unit 
12350  Research  Parkway 
Orlando.  FI.  32826 


8.  PERFORMING  ORGANIZATION  REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Institute  for  the  Behavbral  and  Social 
Sciences 

5001  Eisenhower  Avenue 
Alexandria,  VA  22304-4841 


10.  MONITOR  ACRONYM 

ARI 


11.  MONITOR  REPORT  NUMBER 
Technical  Report  1 141 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 


14.  ABSTRACT  (Maximum  200  words): 

As  the  U.S.  Army  undergoes  transformation,  it  will  need  reliable  means  of  measuring  and  training  complex  new 
skills.  Acquiring  those  will  require  analysis  of  human  behavior  in  the  context  of  human-in-the-loop  simulations  of 
Future  Combat  Systems  (FCS)  still  in  the  concept  exploration  phase.  The  goal  of  the  present  effort  was  to  establish 
and  measure  command  group  behavior  observed  in  such  a  human-in-the-loop  simulation  via  analysis  of  the  verbal 
interactions  of  the  command  group.  A  coding  scheme  for  command  group  verbal  interactions  was  devised  and 
applied  to  several  simulation  runs  in  which  a  command  group  of  4  experienced  Army  Lt.  Col.’s  controlled  an  array  of 
simulated  unmanned  air  and  ground  platforms  to  accomplish  a  deliberate  attack  mission.  This  report  documents  the 
development  of  the  coding  scheme,  the  analysis  results,  and  considers  various  other  approaches  for  analyzing 
verbal  data. 


15.  SUBJECT  TERMS 

FCS,  Verbal  Analysis,  Verbal  Coding,  Command  and  Control,  Battlefield  Functions,  METT-TC,  Robotics,  Future  Force,  C4ISR 


SECURITY  CLASSIFICATION  OF 

19.  LIMITATION  OF 

20.  NUMBER 

21.  RESPONSIBLE  PERSON 

16.  REPORT 
Unclassified 

17.  ABSTRACT 

Unclassified 

18.  THIS  PAGE 
Unclassified 

ABSTRACT 

Unlimited 

OF  PAGES 

Paula  J.  Durlach 

407/384-3983 

1 


11 


Technical  Report  1141 


Coding  Verbal  Interactions  In  A  Prototype  Future  Force 
Command  And  Control  Simulation 


Paula  J.  Durlach 

U.S.  Army  Research  Institute 

Laticia  D.  Bowens  and  John  L.  Neumann 

University  of  Central  Florida 
Consortium  Research  Fellows  Program 

Thomas  J.  Carnahan 

Western  Kentucky  University 
Consortium  Research  Fellows  Program 


Simulator  Systems  Research  Unit 
Stephen  L.  Goldberg,  Chief 


U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue,  Alexandria,  Virginia  22304-4841 


January  2004 


Army  Project  Number 
20262785A790 

Approved  for  public  release;  distribution  unlimited 

iii 


Personnel  Performance 
and  Training  Technology 


IV 


FOREWORD 


As  the  U.S.  Army  undergoes  transformation,  members  of  the  future  force  will  need  to 
acquire  complex  new  skills  in  an  environment  of  rapid  doctrine,  organization,  training,  materiel, 
leadership,  personnel,  and  facilities  changes  and  decreased  deployment  preparation  time.  A 
good  understanding  of  the  required  Command,  Control,  Communications,  Computers, 
Intelligence,  Surveillance  and  Reconnaissance  (C4ISR)  skills  and  a  reliable  means  of  measuring 
and  training  them  will  be  essential.  Acquiring  those  will  require  analysis  of  human  behavior  in 
the  context  of  human-in-the-loop  simulations  of  Future  Combat  Systems  (FCS)  still  in  the 
concept  exploration  phase. 

This  report  documents  the  development  of  a  coding  scheme  for  the  analysis  of  command 
group  verbal  interactions  in  such  a  human-in-the-loop  simulation.  Command  group  members 
were  asked  to  “think  out  loud”  so  that  their  thinking  processes  could  be  captured  among  the 
verbal  interactions.  The  analysis  of  their  verbal  interactions  yielded  a  measurement  instrument 
that  can  be  used  to  help  establish  and  assess  training  of  the  C4ISR  skills  required  by  the  future 
force. 


The  work  described  in  this  report  is  a  portion  of  the  research  performed  under  the  FCS 
Command  and  Control  (C2)  program,  led  by  the  Defense  Advanced  Research  Projects  Agency 
(DARPA)  and  the  U.S.  Army  Communications-Electronics  Command  (CECOM)  Research  and 
Development  and  Engineering  Center  (RDEC).  As  a  program  partner,  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  focused  on  measuring  human  performance 
to  improve  human  system  integration  and  training  and  to  support  the  Science  and  Technology 
Objective  (STO)  titled  “Methods  and  Measures  of  Commander-Centric  Training.” 

Findings  from  this  effort  were  briefed  to  the  Deputy  Chief  of  Staff  for  Operations  and 
Training  from  the  Training  and  Doctrine  Command  (TRADOC  DCSOPS&T).  Methods  and 
findings  from  each  of  the  four  experiments  were  provided  to  the  Program  Manager  (PM)  for  FCS 
C2  as  part  of  ARI’s  ongoing  support  to  FCS  and  Army  research  and  development  efforts. 

Human  performance  findings  by  ARI  helped  shape  the  C2  prototype  showcased  in  the  FCS 
Capstone  Demonstration  of  C2  systems  prior  to  the  FCS  Milestone  B  decision. 


MICHAEL  G.  RUMSEY 
Acting  Technical  Director 
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CODING  VERBAL  INTERACTIONS  IN  A  PROTOTYPE  FUTURE  FORCE  COMMAND 
AND  CONTROL  SIMULATION 

EXECUTIVE  SUMMARY 


Research  Requirement: 

As  the  U.S.  Army  undergoes  transformation,  members  of  the  future  force  will  need  to 
acquire  complex  new  skills  in  an  environment  of  rapid  doctrine,  organization,  training,  materiel, 
leadership,  personnel,  and  facilities  changes  and  decreased  deployment  preparation  time.  A 
good  understanding  of  the  required  Command,  Control,  Communications,  Computers, 
Intelligence,  Surveillance  and  Reconnaissance  (C4ISR)  skills  and  a  reliable  means  of  measuring 
and  training  them  will  be  essential.  Acquiring  those  will  require  analysis  of  human  behavior  in 
the  context  of  human-in-the-loop  simulations  of  Future  Combat  Systems  (FCS)  still  in  the 
concept  exploration  phase.  The  FCS  Command  and  Control  (C2)  program  provides  a  working 
example  of  such  a  simulation.  The  program  leads  are  the  Defense  Advanced  Research  Projects 
Agency  (DARPA)  and  the  U.S.  Army  Communications-Electronics  Command  (CECOM) 
Research  and  Development  and  Engineering  Center  (RDEC).  As  a  program  partner,  the  U.S. 
Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  focused  on  measuring 
human  performance  to  improve  human  system  integration  and  training  and  to  support  the 
Science  and  Technology  Objective  (STO)  titled  “Methods  and  Measures  of  Commander-Centric 
Training.” 


Procedure: 

The  FCS  C2  program  created  a  transformation  environment  for  empirical  assessment  of 
command  group  performance  at  the  small  unit  level,  the  notional  Unit  Cell.  From  October  2001 
to  March  2003,  the  program  conducted  a  series  of  commander-in-the-loop  experiments  at  Ft. 
Monmouth,  NJ.  Over  the  course  of  the  experiments,  the  digital  command  and  control  system — 
the  Commander’s  Support  Environment  (CSE)-became  increasingly  sophisticated,  allowing  the 
command  group  (of  4  people)  to  directly  control  13  simulated  unmanned  air  and  ground  vehicle 
platforms,  and  a  set  of  unmanned  ground  sensors.  The  mission  was  consistently  set  in  simulated 
desert  terrain  from  the  National  Training  Center  (NTC)  and  required  the  Unit  Cell  to  conduct 
deliberate  attack  missions  against  an  enemy  battalion  to  clear  passage  lanes  for  a  follow-on 
force. 


The  ARI  partners  developed  analyses  of  critical  command  group  functions,  and  described 
player  reactions  such  as  workload  assessment,  and  CSE  strengths  and  weaknesses.  The  data  for 
this  work  consisted  of  the  verbal  interactions  among  the  experiment  participants  in  the  Unit  Cell, 
their  computer  interaction  activities,  and  their  responses  to  questioimaires.  The  present  report  is 
concerned  specifically  with  documenting  the  coding  scheme  used  to  analyze  the  verbal 
interaction  data.  Unit  cell  members  were  asked  to  “think  out  loud”  so  that  their  thinking 
processes  could  be  captured  among  the  verbal  interactions.  The  verbal  interactions  of  the  Unit 
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Cell  for  selected  experimental  sessions  were  transcribed  and  content  was  analyzed  for  Function 
(Plan,  See,  Move,  Strike,  Battle  Damage  Assessment),  Source  (who  was  speaking).  Type  of 
communication  (e.g.,  question,  declaration,  etc.),  and  Factor  (Mission,  Enemy,  Terrain,  Troops, 
Time,  Civilians).  As  well  as  documenting  the  coding  scheme  and  presenting  illustrative  results, 
this  report  also  includes  recommendations  for  further  refinement  of  the  coding  scheme. 


Findings: 

A  striking  finding  from  the  analysis  of  verbal  interactions  among  the  Unit  Cell  members 
was  that  sharing  of  information  was  consistently  the  most  frequent  type  of  verbal  interaction  (as 
opposed,  e.g.,  to  questions  or  directions).  Despite  the  fact  that  all  Unit  Cell  members  did  have 
access  to  the  same  database  of  information,  it  was  still  essential  to  verbalize  about  data  that  were 
observed  in  order  to  maintain  a  common  operating  picture.  Analogously,  See  was  always  the 
most  frequently  observed  Fimction  (defined  as  “Detect  or  identify  enemy  or  fnendly  positions,  or 
significant  terrain  aspects”).  This  result  has  significant  implications  for  communications 
requirements  for  the  FCS,  intended  to  be  a  netcentric  system  of  systems  supporting  a  distributed 
force.  Specifically,  one  implication  is  that  a  networked  information  system,  allowing  all  users  to 
access  all  information  may  still  not  be  sufficient  to  maintain  a  common  operating  picture.  Due  to 
the  vastness  of  the  data  available,  and  the  limitations  on  the  human  information  processing 
system,  some  means  of  directing  users  to  the  most  relevant  information  is  going  to  be  required. 

In  the  present  research,  this  was  accomplished  by  participants  telling  one  another  what  they 
believed  they  should  know.  It  should  be  kept  in  mind  however,  that  participants  were  instructed 
to  “think  out  loud.”  This  may  have  distorted  the  type  of  verbal  interactions  observed,  compared 
with  participants  not  so  instructed. 

From  a  methodological  point  of  view,  there  were  several  findings.  A  workable  coding 
scheme  for  analyzing  command  group  verbal  interactions  was  established.  This  scheme  proved 
sensitive  enough  to  discriminate  the  verbal  behavior  of  an  expert  vs.  a  novice  command  group, 
and  to  discriminate  the  verbal  behavior  observed  in  mission  execution  vs.  mission  planning. 

Several  issues  were  tackled,  such  as  how  finely  to  analyze  the  verbal  data  (what 
constitutes  a  data  point?).  The  issues  involved  in  developing  a  coding  scheme  for  verbal  data 
were  explored  and  the  advantages  and  disadvantages  of  different  approaches  were  illustrated.  In 
particular,  we  found  a  trade-off  between  the  granularity  of  analysis  and  the  ability  to  capture  in 
coded  data,  verbalizations  reflecting  command  considerations  and  more  abstract  thinking  and 
decision  making.  In  the  final  conclusion,  it  is  suggested  that  the  most  useful  coding  scheme 
would  be  one  specifically  tailored  to  support  hypothesis-driven  research. 


Utilization  of  Findings: 

This  work  established  a  coding  scheme  for  analyzing  the  verbal  interactions  of  a 
command  group  during  mission  planning  and  execution.  This  measurement  tool  can  now  be 
used  further  to  measure,  help  establish,  and  assess  training  of  the  C4ISR  skills  required  by  the 
future  force. 
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CODING  VERBAL  INTERACTIONS  IN  A  PROTOTYPE  FUTURE  FORCE  COMMAND 

AND  CONTROL  SIMULATION 


The  Experimental  Context 

Between  2001  and  2003,  the  Defense  Advanced  Research  Projects  Agency  (DARPA) 
conducted  a  series  of  experiments  on  Future  Combat  Systems-Command  and  Control  (FCS-C2). 
These  experiments  were  designed  to  assess  the  FCS  concept  that  a  small  group  of  people, 
supported  by  a  digital  command  and  control  system  (C2)  and  several  unmanned  and  some 
manned  platforms,  could  fulfill  the  functions  of  a  traditional  U.S.  Army  command  and  staff 
organization.  In  conjunction  with  the  U.S.  Army  Communications-Electronics  Command 
(CECOM)  Research  and  Development  Center  C^EC),  DARPA  has  conducted  five  warfighter- 
in-the-loop  experiments  focused  on  Command,  Control,  Communications,  Computers, 
Intelligence,  Surveillance,  and  Reconnaissance  (C4ISR)  at  the  Unit  Cell  and  team  level. 

For  four  of  the  experiments,  the  cell  participants  were  the  same  four  U.S.  Army 
Lieutenant  Colonels,  though  on  occasion,  one  or  more  of  them  were  replaced  by  other  players. 
The  remaining  experiment  used  West  Point  Cadets  and  university  ROTC  students.  Four  cell 
positions  were  labeled  as  Commander,  Battlespace  Manager,  Effects  Manager,  and  Intelligence 
Manager.  These  players  sat  in  a  mock-up  C2  vehicle,  each  equipped  with  two  computer  screens, 
which  could  display  information  from  the  digital  C2  system,  the  Commander’s  Support 
Environment  (CSE).  Each  operator  controlled  their  own  displays.  After  Experiment  2,  an 
additional  display  was  added~a  Head  Up  display  that  could  be  seen  by  all  positions.  Any  player 
coiild  put  one  of  their  screens  on  the  head’s  up  display,  in  order  to  facilitate  interactions  and 
discussions  with  the  other  positions. 

One  major  technical  effort  involved  development  of  the  digital  C2  system — the 
Commander’s  Support  Environment  (CSE).  The  CSE  is  defined  as  a  hardware  and  software, 
networked  environment  that  enables  command  and  control  of  the  FCS  Unit  Cell.  It  provided 
visual  displays,  control  of  unmanned  platforms,  computing,  and  communications  infrastructure 
(DARPA,  2001).  Ideally,  the  CSE  provides  the  right  information,  at  the  right  place,  and  at  the 
right  time,  to  enable  the  command  group  to  handle  emerging  conditions  and  accomplish  their 
mission.  Over  the  course  of  the  experiments,  the  CSE  evolved  and  drew  closer  to  this  ideal.  For 
example,  in  Experiment  1,  operators  had  no  ability  to  control  their  own  fires,  but  had  to  call  a 
higher  echelon  to  request  fires.  In  contrast,  by  Experiment  4,  operators  could  not  only  control 
their  own  fires  manually,  but  also  had  an  Attack  Guidance  Matrix  (AGM)  which  would  cause  the 
unmanned  platforms  to  launch  fires  automatically,  according  to  a  set  of  conditions  configured  set 
of  conditions.  Many  of  the  modifications  made  to  the  CSE  over  the  course  of  the  experiments 
were  based  on  feedback  from  the  four  main  participants. 

The  CSE  was  bridged  to  One-SAF  Testbed  (0TB)  to  allow  for  mission  simulation. 
Another  technical  effort  involved  refining  the  behaviors  of  the  simulated  assets  of  the  cell.  The 
configuration  of  assets  included  four  micro-unmanned  aerial  vehicles  (UAVS),  and  a  Shadow 
UAV  (a  large  sized  UAV).  These  were  typically  controlled  by  the  Intelligence  Manager.  There 
.  were  two  line-of-sight  (LOS)  ground  robotic  weapons  platforms,  two  Roboscouts  (groimd 
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robotic  sensors),  and  two  future-warrior  carriers  (FWC),  from  which  computer  generated 
infantry  could  ^  dismounted.  The  Battlespace  Manager  typically  controlled  these  assets,  and 
any  ground  maneuvers.  There  were  two  nonline-of-sight  ^LOS)  ground  robotic  weapons 
platforms  typically  controlled  by  the  Effects  Manager.  Finally,  there  were  two  reconnaissance 
UAVs  (A-160s),  controlled  at  a  higher  echelon,  which  could  be  requested  for  specific 
reconnaissance  missions.  TTie  CSE  included  a  situation  awareness  map  that  displayed  the 
locations  of  these  assets  as  well  as  detected  enemy  or  other  elements,  these  being  displayed  at 
varying  levels  of  confidence  and  identification.  The  CSE  was  used  for  control  of  all  assets  and 
provided  several  aides  to  the  operators  (e.g.,  customized  alerts,  terrain  analysis  features). 

Besides  the  four  Unit  Cell  members  working  with  the  CSE,  after  Experiment  1,  the  C2  vehicle 
itself  also  had  a  human  driver  and  gunner  with  their  own  displays. 

In  addition  to  the  players  in  the  C2  vehicle  (Black  Cell),  there  were  other  cells  that  were 
partly  manned,  and  partly  semi-automated  forces  (S  AF).  The  Blue  Cell  represented  one  echelon 
up  from  black,  and  controlled  the  two  A-160s.  The  White  Cell  represented  blue’s  higher;  white 
did  not  participate  much  in  the  last  few  experiments,  at  least  verbally.  The  Red  Cell  represented 
the  enemy  cell. 

Each  experiment  consisted  of  several  sessions  run  over  two  weeks  time,  with  each 
session  lasting  from  about  30  minutes  to  two  hours.  Shorter  sessions  were  usually  curtailed  due 
to  technical  difficulties;  otherwise  they  were  terminated  according  to  the  judgment  of  the  project 
manager  or  operator/controllers.  During  each  session,  the  cell’s  mission  was  similar:  to  traverse 
from  one  position  to  another,  and  secure  the  passageway  and  second  position  with  the  intention 
of  ensuring  safe  passage  of  a  second  (theoretical)  follow-on  Unit  Cell.  Each  session  began  with 
the  delivery  of  intelligence  information  to  the  cell,  a  planning  phase  in  which  they  planned  the 
mission,  and  an  execution  phase  in  which  the  mission  was  conducted.  For  Experiments  1-3,  the 
strength  of  the  opposition  forces  was  systematically  varied  across  sessions.  After-action  reviews 
(AAR)  were  conducted  for  each  session  by  expert  analysts. 

Prior  to  each  experiment,  participants  underwent  varying  amounts  of  (imstructured) 
training,  to  familiarize  them  with  CSE  operations  and  any  upgrades  that  had  been  made  since  the 
previous  experiment.  The  amoirnt  of  training  was  subject  to  program  technical  progress  and  the 
translation  of  this  into  training  materials. 

U.S.  Army  Research  Institute  Role 

The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI) 
(Armored  Forces  Research  Unit  at  Ft.  Knox  and  Simulator  Systems  Research  Unit  at  Orlando) 
supported  this  research,  by  conducting  analysis  of  the  participants’  behavior  during  the 
experiments,  and  their  reactions  to  their  experience.  From  ARI’s  perspective,  this  research 
contributes  to  the  Science  and  Technology  Objectives  (STO),  Methods  and  Measures  of 
Commander-Centric  Training.  As  part  of  that  STO  effort,  ARI  i:  investigating  the  requirements 
for  training  army  officers  to  perform  complex  C4ISR  activities  that  encompass  multiple 
traditional  military  fimctions.  The  FCS  concept  entails  an  xmprecedented  alliance  of  humans  and 
machines  at  the  small  unit  level  and  the  C4ISR  behaviors  of  the  command  group  for  an  FCS  Unit 
Cell  need  to  be  investigated  and  understood.  The  ARI  research  developed  detailed  analyses  of 
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critical  command  group  flmctions,  and  measured  player  reactions  such  as  workload  assessment, 
and  CSE  strengths  and  weaknesses.  The  data  for  this  work  consisted  of  the  verbal  interactions 
among  the  Black,  Blue,  and  White  cells,  questionnaire  responses  from  members  of  the  Black 
cell,  and  human-computer  interaction  activities.  The  ARI  research  for  this  project  is  reported  in 
detail  in  the  reports:  PCS  C2  Human  Functions  Assessment:  Interim  Report,  for  Experiment  1 
(Sanders,  Lickteig,  Durlach,  Rademacher,  Holt,  Rainey,  Finley,  and  Lussier,  2002),  for 
Experiment  2  (Lickteig,  Sanders,  Durlach,  Rainey,  and  Carnahan,  2002),  for  Experiment  3 
(Lickteig,  Sanders,  Durlach,  and  Carnahan,  2003),  and  for  the  student  research  (Carnahan, 
Lickteig,  Sanders,  and  Durlach,  in  preparation).  The  present  report  is  concerned  with 
documenting  the  development  of  the  coding  scheme  used  to  analyze  the  verbal  data.  Results  will 
be  presented  for  illustrative  purposes.  Readers  wishing  more  detailed  data  on  the  verbal  analysis, 
or  on  the  other  measures  examined  are  referred  to  the  reports  listed  above. 

Development  of  a  Coding  Scheme  for  Verbal  Interaction  Data 

This  report  describes  the  development  of  the  coding  scheme  for  analysis  of  the  verbal 
interactions  recorded  during  selected  sessions  over  the  course  of  the  four  experiments  with  the 
primary  players.  Verbal  data  are  recognized  as  a  window  on  the  psychological  mechanisms  and 
knowledge  structures  underlying  problem-solving  activities  (Ericsson  and  Simon,  1993;  Yang, 
2003).  During  these  experiments,  players  were  encouraged  to  “think  out  loud”  and  indeed, 
players  did  speak  during  approximately  96%  of  each  experimental  session.  Written  transcripts 
were  compiled,  using  audio  recordings  of  specific  sessions.  These  transcripts  served  as  the  basic 
data  for  analysis.  The  goal  of  the  analysis  was  to  determine  and  characterize  the  C4ISR 
behaviors  of  the  Unit  Cell. 

At  a  high  level,  the  function  of  command  group  actions  is  to  C2  the  Unit  Cell  and 
accomplish  the  assigned  mission.  A  candidate  set  of  functions  adopted  initially  was: 

•  Plan:  Develop,  assess,  and  modify  a  plan  including  combat  instmction  sets  provided  to 
robotic  elements  in  response  to  changing  events. 

•  Move:  Control  the  movement  and  activity  of  friendly  maimed  and  unmanned  systems  to 
maintain  desired  movement  rates  and  formations. 

•  See:  Control  and  interpret  input  from  a  heterogeneous  set  of  advanced  sensors  to 
mentally  constmct  an  accurate  picture  of  enemy  activity  and  intent. 

•  Strike:  Distribute  a  variety  of  indirect  and  direct  effects  over  a  set  of  targets. 

The  challenge  was  to  relate  these  functions  to  the  observed  verbal  data,  characterize  their 
relative  frequency  and  time  duration,  and  elaborate  the  verbal  classification  scheme  further.  To 
facilitate  elaboration,  the  team  reviewed  the  draft  CSE  Functions  Manual  (DARPA,  2001), 
which  identified  equipment  functions  of  CSE  system  operation.  The  team  also  reviewed  the 
literature  regarding  C2  group  functions  and  task  classification  strategies,  and  identified  several 
candidate  C2  activities  to  consider  for  use  in  coding  Experiment  1  verbal  communications  data. 
The  coding  scheme  arrived  at  coded  for: 

•  Source:  Who  was  involved  in  the  verbal  interaction. 

•  Function:  Plan,  See,  Move,  Shoot. 
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•  Factor:  21  subfactors  assigned  to  one  of  the  following  larger  categories  of  Mission, 
Enemy,  Terrain,  Troops,  Time,  and  Civilians  (METT-TC). 

In  order  to  apply  this  coding  scheme,  the  team  decided  to  break  each  session  transcript 
into  “chunks”  of  words,  with  each  chunk  addressing  a  single  topic  (as  recommended  by  Ericsson 
and  Simon,  (1993).  It  was  intended  that  one  code  for  each  of  Source,  Fimction,  and  METT-TC 
factor  be  assigned  to  each  chunk.  Ideally,  the  code  to  assign  for  each  block  should  be 
unambiguous.  This  requires  both  that  the  coding  categories  are  clearly  defined  (what  does  a 
particular  code  mean),  and  that  each  chunk  only  have  one  code  appropriate  to  its  content. 
Unfortunately,  human  verbal  interaction  is  not  simple.  People  do  not  interact  in  a  strictly  linear 
way,  and  discussions  don’t  necessarily  involve  one  and  only  one  topic  (Y ang,  2003).  This  means 
that  the  chunking  of  the  dialog  itself  involves  important  assessment  on  the  part  of  the  researcher 
as  to  when  one  chunk  should  end  and  anotiier  begin.  The  way  in  which  the  raw  transcript  is 
chunked  will  influence  how  unambiguously  the  coding  scheme  can  be  applied,  and  in  particular, 
whether  independent  raters  agree  on  the  codes  to  assign. 

The  level  of  analysis  to  examine  behavior  is  a  perpetual  issue  in  psychological  science. 
When  looking  at  behavior  from  a  “high”  level,  it  may  be  too  complex  to  apply  quantitative 
measures  to,  or  be  o^n  to  several  different  interpretations,  such  ^t  the  objectivity  of  the 
analysis  can  be  questioned  (e.g.,  the  quality  of  the  performance  of  an  orchestra).  Looking  at 
behavior  from  a  detailed  molecular  level  tends  to  lend  itself  more  to  quantitative  analysis  and 
increasing  objectivity,  but  this  can  also  be  at  the  loss  of  “the  big  picture”  (will  analyzing  the 
individual  finger  movements  of  each  musician  help  the  researcher  understand  the  quality  of  the 
orchestral  performance?).  The  question  of  how  finely  to  chunk  session  transcripts  has  been  an 
issue  throughout  this  work.  For  Experiment  1,  chunks  consisted  of  fairly  large  pieces  of  dialog. 
One  ARI  researcher  and  one  Active  Duty  Army  Major  independently  coded  the  chunks  for  each 
of  the  nine  sessions,  compared  their  results,  and  then  reached  agreement  on  a  single  set  of  codes 
for  each  chunk.  To  develop  an  empirical  estimate  of  inter-coder  agreement  the  ARI  researcher 
and  the  first  author  coded  four  session  transcripts  independently  and  compared  their  results. 

There  was  rather  poor  inter-coder  agreement  (74%  for  Source,  63%  for  Function,  and  55%  for 
METT-TC).  The  goal  was  at  least  80%.  The  main  problem  seemed  to  be  that  the  chunks  were 
too  large,  and  that  more  than  one  code  could  be  applied  to  each  one.  When  two  or  three  codes 
were  potentially  applicable,  it  was  then  a  matter  of  subtle  interpretation  as  to  which  one  of  those 
a  particular  coder  selected.  It  was  clear  that  for  subsequent  experiments,  the  chunks  needed  to  be 
more  fine-grained  and  that  guidelines  for  chunking  needed  to  be  developed.  A  sample  of 
unchunked  and  uncoded  transcript  can  be  fmmd  at  the  end  of  this  report. 

The  revised  coding  scheme  developed  was  based  both  on  these  considerations  and  the 
substantive  content  of  the  results  of  Experiment  1  (i.e.,  what  it  was  that  the  players  actually 
talked  about).  The  essence  of  this  scheme— the  common  elements  that  were  used  for  Experiments 
2-4 — ^is  outlined  in  Tables  1-4.  Over  the  course  of  Experiments  2-4  some  additional  categories 
were  used  as  well;  these  will  be  discussed  subsequently.  There  were  four  codes  assigned  to  each 
chunk,  one  from  each  of  the  code  categories  Type,  Source,  Function,  and  Factor. 

Interaction  type  (e.g.,  commands,  questions,  observations)  to  categorize  verbal  interaction 
has  proven  a  useful  predictive  variable  of  team  performance  (Kanki,  Folk,  and  Irwin,  1991).  For 
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the  present  work,  the  Type  category  (see  Table  1)  was  introduced  primarily  to  guide  chunking. 
Chunks  were  initially  determined  by  segmenting  the  verbal  data  into  passages  of  different 
interaction  types.  Share,  Action,  and  Direction  were  relatively  short  interactions,  without  a  lot  of 
discussion.  Chunks  including  a  lot  of  discussion  or  consideration  of  multiple  aspects  of  a 
situation  were  designated  either  Process  or  Decide.  These  were  distinguished  by  whether  there 
was  a  definite  conclusion  reached  by  the  participants  (Decide)  or  not  (Process). 

The  raw  transcripts  were  initially  chunked  according  to  Type  by  one  coder  (the  first 
author).  This  resulted  in  finer-grained  chunks  than  in  Experiment  1;  however,  it  still  led  to 
several  chunks  for  which  multiple  METT-TC  subfactor  codes  were  applicable.  Therefore,  a  two- 
stage  process  of  chunking  was  used.  After  the  raw  transcript  was  chunked  according  to  Type,  on 
a  second  pass,  chunks  were  further  subdivided  if  multiple  METT-TC  subfactor  codes  could  be 
applied  to  the  chunk.  In  addition,  overlapping  interactions  (e.g.,  two  pairs  of  people  having 
separate  conversations  at  the  same  time)  were  separated,  when  intelligible,  such  Aat  each 
conversational  stream  became  a  separate  chunk.  For  Experiments  2-4,  the  chunking  was  always 
carried  out  by  the  same  person  (the  first  author),  and  no  attempt  was  made  to  assess  whether 
other  coders  agreed  with  the  chunking. 

The  remaining  coding  categories  were  Source,  Function,  and  METT-TC  Factor  as  laid 
out  in  Tables  2, 3,  and  4.  Source  coded  for  the  participants  in  the  verbal  chunk.  Function  coded 
for  the  broad  C2  function  under  discussion.  In  addition  to  the  Plan,  See,  Move,  and  Shoot  codes 
(now  relabeled  Strike),  as  used  in  Experiment  1,  two  additional  Function  codes  were  added: 

Battle  Damage  Assessment  (BDA),  and  Other.  BDA  was  added  as  it  seemed  a  very  important 
issue  for  the  cell,  which  should  be  coded  as  a  separate  item.  Because  missions  were  conducted 
from  a  stand-off  position,  progress  of  each  mission  depended  bn  obtaining  reliable  BDA.  The 
code  “Other”  of  Ae  Function  category  was  added  to  capture  any  other  activities  that  did  not  fit 
with  the  specified  functions.  METT-TC  Factor  category  coded  for  traditional  Army  information 
categories.  Under  the  factors  of  Mission,  Enemy,  Terrain,  Troops,  Time,  and  Civilians,  25  sub¬ 
factors  were  specified  to  capture  the  detailed  information  discussed  during  each  session.  Each 
chunk  was  assigned  one  of  the  25  subfactor  codes,  and  then  related  the  hffiTT-TC  factors,  as  laid 
out  in  Table  4. 


Table  1 .  The  Type  Category  used  for  coding  Experiments  2-4 


1 

Share.  Announcement,  telling  what  is  seen  or  known. 

2 

Action.  Annoimcement.  telline  what  speaker  is  doing  at  the  moment-verbalization 
accomnanvine  action  such  as  fire  or  move.  Not  the  decision  process.  Not  actions 
such  as  I  see,  monitor,  track,  etc.  Not  describing  someone  else’s  actions. 

Direction.  Order,  command,  delegation  of  responsibility. 

■ 

Ask.  Interaction  beeins  with  request  for  information,  confirmation,  assistance,  or 
assets  and  ends  either  with  informational  answer  or  no  resnonse.  with  little  or  no 
discussion.  Not  rhetorical  questions. 

5 

Process.  Infer,  synthesize,  fuse,  understand,  turn  data  into  information  without 
consequent  decision  or  direction.  Can  start  with  Share,  Action,  or  Ask. 

Decide.  Like  Process,  but  in  addition,  includes  a  verbalized  decision  or  plan. 

7 

Other. 
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Table  2.  The  Source  Category  usee 

for  coding  Experiments  2-4 

1 

Within  Cell  (Black) 

Cell  =  4  CSE  operators 

2 

Cell  <->  Blue  (Team) 

3 

Cell  <->  White  (Higher) 

tm 

Cell<->Subordinate 

Subordinate  includes  C'^Vehicle  gunner  &  driver 

Blue<->  White 

6 

More  than  2-way  (e.g., 
Cell<->White<->Blue) 

Only  to  be  used  in  cases  where  more  than  2  elements 
involved  in  SAME  conversation 

Other 

E.g.,  to  technical  support  people 

Table  3.  The  Function  Category  used  for  coding  Experiments  2-4 


1 

See 

Detect  or  identify  enemy  or  fiiendly  positions,  or 
significant  terrain  aspects,  (not  BDA) 

2 

Plan 

Interpret  data,  predict  enemy  CO  A,  generate  own 
COA 

3 

Move 

Manage/Monitor/Control  asset  movement 

4 

Strike 

Manage/Monitor/Control  lethal/nonlethal  effects 

5 

BDA 

See  for  purposes  of  BDA 

6 

other 

None  of  the  above 

In  order  to  assess  the  objectivity  of  the  coding  scheme,  results  were  assessed  for  inter¬ 
coder  agreement.  First,  two  coders  (the  first  author  and  the  fourth  author)  coded  a  single  session 
from  Experiment  2  independently,  and  then  discussed  their  codes  for  each  chunk,  focusing  on 
those  chunks  for  which  their  codes  differed.  Nuances  in  interpretation  were  debated,  and 
through  this,  a  common  understanding  on  how  to  apply  the  codes  was  achieved.  The  two  coders 
subsequently  coded  two  additional  sessions  and  their  probability  of  agreement  for  each  session 
was  calculated,  as  shown  in  Table  5.  As  can  be  seen  in  Table  5,  the  level  of  agreement  was 
relatively  good,  and  definitely  an  improvement  over  Experiment  1 .  For  Experiments  3  and  4,  the 
coding  was  conducted  by  one  coder  only  (the  first  author). 

It  should  be  noted  that,  for  Experiment  2,  we  also  tried  a  more  automated  form  of 
analysis,  by  using  key-word  or  phrase  searches  and  counts.  For  example,  by  counting  the 
number  of  times  the  phrase  “I  think”  occurred  and  other  such  phrases,  it  might  be  possible  to 
arrive  at  a  quantitative  measure  of  the  number  of  situation  assessments  made  by  the  cell.  In 
practice,  however,  it  turned  out  that  there  were  many  times  that  key-phrases  were  used  in  a  way 
that  had  little  or  nothing  to  do  with  their  intended  interpretation,  as  well  as  instances  that  clearly 
did  fit  with  the  intended  interpretation  but  without  mention  of  a  key  phrase.  Thus,  this  approach 
did  not  really  yield  useful  data  in  and  of  itself.  It  is  possible  that  now  that  more  transcripts  have 
been  coded,  a  re-analysis  of  the  phrases  used  most  commonly  for  each  code  set  might  reveal 
more  successful  search  targets. 
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Table  4.  The  METT-TC  Factor  Category  used  for  coding  Experiments  2-4 


MISSION 


1 

Original  Plan:  Concerning  mission  goals  and  plans  prior  to  execute  phase. 

2 

Dynamic  Planning:  Tactical  re-planning  during  the  execute  phase  in  response  to 
changing  events  and  available  assets.  Must  have  stated  CO  A  (course  of  action). 
Changes  from  Original  Plan. 

Situational  Understanding.  Integration/summary  of  current  situation  involving 
multiple  factors;  but  vrithout  stated  COA. 

ENEMY :  Concerning  enemy  situation  including 

um 

Location:  Sensor  hit(s)  -  locate  enemy  positions. 

Identification:  Identify  targets  -  identify  nature  of  enemy  target. 

Disposition:  Probable  enemy  COA,  strategy,  or  tactics. 

BDA:  Battle  Damage  Assessment  -  cell  seeks/discusses  feedback  on  damage  they 
inflict  on  enemy. 

TERRAIN 

8 

When  terrain  is  the  prime  focus  (e.g.,  can  we  travel  over  that  kind  of  terrain?,  we 
should  go  this  way  because  it  will  provide  cover).  Example:  “Moving  to  low 
ground.”  Not  simply  map  locations  (e.g.,  not,  sensor  hit  north  of  the  wall). 

TROOPS  and  Assets  (Soldiers,  Equipment,  Vehicles) 

FRIENDLY  ONLY 

9 

Location  Status:  Position  report/assessment 

10 

Movement  Status:  Mobility  report/assessment  (includes  fuel) 

11 

See  Status:  Sensor  report/assessment 

12 

Strike  Status:  Fire  power  report/assessment  (includes  #  of  remaining  missiles) 

Communications/network  functionality  (radio,  internet,  or  other;  cell  to  outside  cell, 
including  semi-autonomous  sensors). 

14 

Information  management  systems:  CSE  user  interface  tools 

15 

Survivability  Concern;  Asset  in  danger. 

16 

Survivability  Move:  defensive  move  to  remove  asset  from  immediate  danger. 

17 

Loss/Casualty:  Asset  destroyed  (catastrophic  hit). 

18 

Move  Action:  Move/Manage/Maneuver  [Active,  Not  position  report] 

Excluding  Survivability  Move;  Also  See  Terrain. 

19 

Strike  Action  Lethal:  Laxmch/fire/deploy  with  intent  to  destroy  (includes  LAMs) 

20 

Strike  Action  Nonlethal:  Laimch/fire/deploy  (could  include  unarmed  sensors, 
propaganda,  smoke,  jamming  of  enemy,  etc.). 

21 

Training  (Soldier  training,  mission  rehearsal) 

22 

Other—  having  to  do  with  troops  or  assets  but  none  of  the  above 

TIM 

23 

^en  time  is  the  prime  focus  (e.g.,  how  much  time  something  will  take,  how  much 
time  is  available,  order  of  priority,  synchronization  of  actions). 

CIV] 

[LIANS 

24 

Any  issues  regarding  how  to  deal  with  civilians:  avoiding,  provisioning,  protecting, 
etc.  Not  mere  sensor  hits  of  civilians,  unless  first  time  mentioned. 

OTF 

lER 
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Other  (e.g.,  humor,  personal,  leadership,  morale) 

Table  5.  Probability  of  inter-coder  agreement  for  2  sessions  of  Experiment  2 


Session 

Number 

Type 

Source 

Fimction 

Factor 

6 

.94 

.99 

.94 

.89 

8 

.87 

.99 

.92 

.86 

Chance  ♦ 

.14 

.14 

.17 

.04 

♦Probability  of  chance  agreement  based  on  number  of  potential  codes  for  the  category. 

Results  of  Experiments  2-4 

To  illustrate  the  substance  of  the  results  obtained,  the  data  from  one  session  from  each  of 
Experiments  2,  3,  and  4  will  be  presented.  It  should  be  remembered  that  these  data  reflect  only 
what  was  verbalized  by  the  participants.  Low  values  on  some  of  the  categories  don’t  necessarily 
indicate  that  those  activities  were  not  engaged  in  frequently,  only  that  they  were  not  talked  about 
frequently.  A  good  example  is  the  Action  code,  of  the  Type  category.  The  Intelligence, 
Battlespace,  and  Effects  managers  were  all  quite  active  using  the  CSE  controls  throughout  each 
session;  however,  they  did  not  necessarily  report  on  all  their  activities  to  the  rest  of  the  group. 

Table  6  illustrates  some  key  characteristics  of  the  three  sessions.  As  can  be  seen  in  Table 
6,  the  three  sessions  were  of  approximately  equal  duration,  with  verbal  data  collected  during 
more  than  95%  of  each  session.  On  average,  each  chunk  lasted  1 1.4  seconds,  and  consisted  of 
26.7  words. 


Table  6.  Key  Characteristics  of  the  verbal  data 


Experiment  2 

Experiment  3 

Experiment  4 

Run  Duration 

87  minutes 

89  minutes 

90  minutes 

Cumulative  Silence^ 

2.3  minutes 

2.9  minutes 

3.75  minutes 

#  Verbal  Chunks 

494 

461 

408 

#  Words  in  transcript 

11,984 

12,329 

11,848 

♦Timing  initiated  after  3  seconds  of  silence 


Figure  1  illustrates  the  verbal  data  coded  for  Type  of  interaction,  and  Table  7  lists 
selected  chunks  to  illustrate  each  Type  code.  As  can  be  seen  in  the  figure,  the  distribution  of 
chunks  across  the  Type  codes  was  very  similar  across  experiments.  The  most  frequent  type  of 
interaction  was  always  Share  (an  announcement  telling  what  is  seen  or  known).  Share 
interactions  tended  to  be  relatively  brief.  Their  frequency  indicates  the  importance  of  the  cell 
members  keeping  one  another  “on  the  same  page.”  Despite  the  fact  that  they  shared  the  same 
digital  C2  system,  each  operator  configured  their  system  in  a  unique  way,  such  that  they  were 
rarely  all  looking  at  the  same  display  configuration.  Their  use  of  the  head’s  up  display  (not 
captured  in  the  coding  scheme),  indicates  the  requirement  to  refer  to  a  common  picture.  That 
head’s  up  display  was  actually  requested  by  the  operators  after  Experiment  1.  The  feeling  of  the 
need  for  a  head’s  up  display,  and  the  frequent  Share  verbalizations  suggest  that  provision  of  a 
common  digital  C2  system  is  not  sufficient  to  maintain  a  common  operating  picture. 

Next  most  frequent  were  Ask  and  Direction,  with  their  relative  frequency  dependent  on 
the  particular  session.  These  were  also  fairly  brief  interactions.  For  example,  Ask  typically 
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involved  a  request  for  some  information,  followed  by  a  straight  forward  answer.  Direction 
typically  involved  a  request  for  some  action  to  be  performed  and  an  acknowledgement.  Least 
frequent  were  Action,  Process,  and  Decide.  Process  and  Decide,  by  definition,  tended  to  be 
relatively  longer  chunks  with  multiple  interactions.  Their  low  frequency  does  not  necessarily 
imply  that  the  cell  paid  insufficient  attention  to  thinking  things  through.  A  clear  relation  betw'een 
mission  performance  and  Type  frequency  would  be  necessary  to  demonstrate  this. 

Unfortunately,  the  ability  to  relate  the  verbal  data  to  mission  performance  was  not  possible,  as 
mission  performance  assessment  was  not  available  for  analysis  (there  were  AAR  sessions,  but 
the  data  from  these  were  not  captured  formally  for  later  analysis). 


□  E2  □  E3  ■  E4 
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Figure  1 .  Relative  percent  of  verbal  interactions  assigned  each  type  code,  for  one  session  from 
each  experiment  (E2-E4). 
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Table  7.  Example  chunks  for  each  Type  code.  Note:  these  are  not  consecutive  chunks 
Speaker  T  Verbal  Data  |  Type 


Intelligence 

Manager 

There's  a  target  that  popped  up  there  from  our  Elint. 

I  believed  that  popped  up  from  our  Elint-this  Daria. 

Let  me  just  find  out  a  little  more  information  on  this. 

Share 

Commander 

Alright,  I'm  firing  two  UGS  with  netfires  1 

Action 

Commander 

Alright,  here’s  what  I  want  you  to  do,  Ken. 

Direction 

Intelligence 

Manager 

Yes,  sir. 

Commander 

I  want  to  fly  that  SAR  path  with  the  A- 160.  Then,  when  it  gets 
done  painting  that  northern  sector,  turn  it  back  to  MTI.  Then 
turn  our  SAR-Bird  back  on  and  put  SAR-Bird  back  in  on 
yellow.  Okay? 

Commander 

Where’s  my  shadow  at? 

Ask 

Battlespace 

Manager 

He’s  moving... the  MTL track  has  him  moving  east. 

Process 

Commander 

He’s  going  to  go  around.  I  think  what  he  is  going  to  do  is  go 
around  that  hill  mass  and  come  down  by  Karen. 

Commander 

Blue6-Black6 

Decide 

Blue 

Blue.  Over. 

Commander 

Alright  Blue6-Black6.  Initial  read  of  the  battlefield  is  that  he 
is  defending  forward  with  a  heavy  security  zone  established 
along  the  PLOT.  Break.  I  detect  at  least  2  platoons  forward 
along  the  PLOT  supported  by  air  defense  artillery.  Break. 
Additionally,  it  looks  like  he's  also  got  some  concentration  in 
the  center.  Therefore  I  suspect  a  heavy  forward  defense,  a 
heavy  security  zone  with  a  mobile  defense  more  than  likely 
coming  from  the  center  sector.  Break.  Therefore  1  think  I've 
got  all  the  information  I  need  to  begin  executing  decision 
point  1  and  our  attack  in  the  north.  Over. 

Blue 

Roger,  I  think  you'll  find  more  by  Granite  Pass,  on  the  back 
side  of  Iron  Triangle.  I  think  we  have  a  good  read.  Over. 
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Table  8  shows  the  verbal  data  for  each  session,  coded  for  the  Source  category.  Clearly  the 
vast  majority  of  verbal  interactions  were  among  the  Black  Unit  Ceil  managers.  The  next  most 
frequent  interactions  were  between  the  Black  cell  and  their  driver  or  gunner  (subordinates)  and 
between  the  Black  cell  and  the  Blue  cell.  Interactions  listed  as  other  typically  involved  members 
of  the  Black  cell  and  either  technicians  supporting  the  simulation  or  visitors  observing  the 
simulation  from  within  the  C2  vehicle  room. 


Table  8.  Mean  percent  of  verbal  interaction  observed  for  each  code  of  Source  category 


Source  code 

Experiment  2 

Experiment  3 

j— - _ 

Experiment  4 

Within  Black  Cell 

84.0 

86.1 

84.6 

Black  - 
Subordinates 

7.9 

3.7 

10.0 

Black  -  Blue 

5.1 

4.6 

3.9 

Black  -  White 

0.8 

0.0 

0.0 

Blue  -  White 

1.2 

3 -way 

Other 

5.6 

1.5 

Figure  2  illustrates  the  verbal  data  categorized  according  to  the  Function  codes.  As  can 
be  seen  in  Figure  2,  the  distribution  of  Function  across  the  different  codes  was  quite  similar 
across  the  three  sessions  (and  all  the  mission  execution  sessions  that  were  analyzed).  See  was 
always  the  most  frequently  verbalized  Function,  followed  by  Move  or  Strike,  depending  on  the 
particular  session.  This  was  followed  by  BDA  and  then  Plan  and  Other. 


□  E2  □  E3  ■  E4 


Figure  2.  Relative  percent  of  verbal  interactions  assigned  each  function  code,  for  one  session 
from  each  experiment  (E2-E4). 
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The  predominance  of  See  reinforces  the  comments  that  were  made  previously  about 
Share.  That  is,  despite  all  managers  having  the  same  CSE  system,  there  was  a  strong  tendency  to 
verbally  communicate  with  one  another  concerning  what  was  observed. 

There  appears  to  be  a  trend  over  experiments,  for  the  relative  frequency  of  Move  to 
decrease  and  the  relative  frequency  of  BDA  to  increase.  This  may  have  resulted  from  continual 
refinements  to  the  CSE  over  the  course  of  experiments.  Firstly,  over  the  experiments,  tasking  of 
unmanned  platforms  became  more  reliable,  and  the  tasks  that  could  be  assigned  to  the  unmanned 
platforms  more  sophisticated  (e.g,,  from  each  platform  needing  to  be  tasked  individually  to 
collective  tasking  such  as  “group-follow”).  Therefore,  as  the  ability  to  move  platforms  became 
increasingly  better,  the  need  to  talk  about  platform  movement  may  have  decreased.  Secondly,  as 
the  importance  of  good  BDA  data  became  clear  in  the  operational  context,  more  effort  was  made 
by  the  cell  to  check  on  the  results  of  their  strikes.  In  addition,  by  Experiment  4,  some  weapons 
systems  were  equipped  with  automatic  “Chasers:”  sensors  that  were  automatically  sent  out  after 
a  strike,  to  collect  BDA  information.  Such  factors  probably  led  to  the  gradual  increase  in  the 
number  of  interactions  concerning  BDA.  Table  9  gives  example  chunks  for  each  Fxmction  code. 

Figure  3  shows  the  distribution  of  verbal  data  over  the  codes  for  METT-TC  Factor.  As 
can  be  seen  in  the  figure,  the  vast  majority  of  verbal  chunks  were  coded  as  Enemy  or  Troops. 
Therefore,  these  categories  are  illustrated  across  their  subfactors  in  Figures  4  and  5,  respectively. 
With  regard  to  the  Enemy  Factor,  the  relative  frequency  of  Locate,  Identify,  and  BDA  were 
approximately  the  same,  with  discussion  of  enemy  Disposition  (probable  enemy  course  of  action 
strategy,  or  tactics)  lower.  With  regard  to  the  Troops  Factor,  chunks  concerning  Strike  and 


Figure  3.  Relative  percent  of  verbal  interactions  assigned  each  factor  code,  for  one  session  from 
each  experiment  (E2-E4). 
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Move  predominated,  followed  by  CSE  issues,  and  status  reports  (e.g.,  sensors,  mobility, 
position).  Other  codes  were  used  rather  infrequently.  Table  10  gives  example  chunks  for  the  4 
Enemy  subfactors.  Table  1 1  gives  example  chunks  for  the  more  frequently  used  Troops 
subfactors. 


Table  9.  Example  chunks  for  each  Function  code.  Note:  these  are  not  consecutive  chunks 
_ Speaker _ Verbal  Data  Function 


Commander 

Blue6-Black6 

Plan 

Blue 

Blue.  Over. 

Commander 

Alright  Blue6-Black6.  Initial  read  of  the  battlefield 
is  that  he  is  defending  forward  with  a  heavy  security 
zone  established  along  the  PLOT.  Break.  I  detect  at 
least  2  platoons  forward  along  the  PLOT  supported 
by  air  defense  artillery.  Break.  Additionally,  it  looks 
like  he's  also  got  some  concentration  in  the  center. 
Therefore  I  suspect  a  heavy  forward  defense,  a 
heavy  security  zone  with  a  mobile  defense  more 
than  likely  coming  from  the  center  sector.  Break. 
Therefore  I  think  I've  got  all  the  information  I  need 
to  begin  executing  decision  point  1  and  our  attack  in 
the  north.  Over. 

Blue 

Roger,  I  think  you'll  find  more  by  Granite  Pass,  on 
the  back  side  of  Iron  Triangle.  I  think  we  have  a 
good  read.  Over. 

Battlespace 

Manager 

That's  a  series  of  movers  down  there.  There's 
wheeled  Unknown  24,  wheeled  HOI,  MTI. . . 

See 

Commander 

OK,  C2  Driver 

Move 

Driver 

Roger 

Commander 

I  want  you  to  pull  over  for  just  a  minute.  There's  a 
LOS  coming  behind  you.  Let  the  LOS  pass  you,  and 
then  I  want  you  to  follow  the  LOS.  How  copy? 

Driver 

Wilco 

Commander 

Roger 

Commander 

Double  tap,  I  want  him  double  tapped.  Brooks,  you 
take  him  too. 

Strike 

Commander 

SA-15  smoking,  change  its  state  please,  gentlemen. 

BDA 

Effects  Manager 

It’s  a  beautiful  thing. 
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Table  10.  Example  chunks  coded  as  Enemy  Factor.  Note:  these  are  not  consecutive  chunks 
_ Speaker _ Verbal  Data _ Subfactor 


Battlespace 

Manager 

Unknown  9,  center  sector,  MTI  SAR 

Locate 

What  does  this  look  like,  Brooksie? 

Identify 

Battlespace 

Manager 

It  looks  like  a  Dreaga  going  up  the  hill. 

-  . - . 

Intelligence 

Manager 

Alright 

Battlespace 

Manger 

It’s  a  tracked  vehicle 

(speaker  unknown) 

Now  he’s  staying  back 

Commander 

Ok  this  tells  me  he  is  defending  forward  along  the 
PLOT. 

Disposition 

Commander 

SA-15  smoking,  change  its  state  please,  gentlemen. 

BDA 

Effects  Manager 

It’s  a  beautiful  thing. 

Figure  4.  Relative  percent  of  verbal  interactions  assigned  across  subcodes  of  the  enemy  factor, 
for  one  session  from  each  experiment  (E2-E4). 
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Table  11.  Example  chunks  coded  as  Troops  Factor.  Note:  these  are  not  consecutive  chunks 
Speaker  |  Verbal  Data  |  Subfacf^ 


Commander 

Double  tap,  I  want  him  double  tapped.  Brooks,  you 
take  him  too. 

Strike 

Commander 

OK,  C2  Driver 

Move 

Driver 

Roger 

Commander 

1  want  you  to  pull  over  for  just  a  minute.  There's  a 
LOS  coming  behind  you.  Let  the  LOS  pass  you,  and 
then  I  want  you  to  follow  the  LOS.  How  copy? 

Driver 

^wiico  ■  - 

Commander 

.  Roger 

Comm^der____  Hey  woah,  I've  lost  my  cursor. _  IT/CSE 


Intelligence 

Manager 

The  resolution  is  very,  very  small.  And,  you  know,  I 
don't  think  I'll  be  able  to  see  any  of  this  at  all. 

Sensors 

Commander 

Are  PAMs  going  in  on  all  4  of  those  targets.  David? 

Strike 

Effects  Manager 

I've  still  got  plenty  of  PAMs.  Don't  worry  about  it.  I 
got  it. 

Ability 

Commander 

Alnght,  I'm  firing  two  UGS  with  netfires  1 

Strike 

Nonlethal 

Battlespace 

Manager 

Everybody  else  is  still  healthy  and  moving 

Mobility 

White  cell 

Black,  White,  Your  push 

Caution 

Commander 

Roger,  White  1 

White  cell 

Black,  Your  northern  most  future  warrior  carrier 
receiving  artillery  fire  now.  Out. 

Commander 

G2V.  Artillery! 

Survivability 

Move 

Battlespace 

Manger 

C2V.  Move  back  to  the  east. 

Commander 

You’ve  got  to  move  him. 

Battlespace 

Manager 

C2V  has  got  to  move  with  him. 
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Figure  5.  Relative  percent  of  verbal  interactions  assigned  across  subfactors  of  the  troops  factor, 
for  one  session  from  each  experiment  (E2-E4).  strikeN  =  strike  nonlethal,  strikeL  —  strike  lethal, 
survMove  =  survivability  move,  comms  =  communications. 

Other  Approaches  Explored 


Systems 

Another  category,  not  mentioned  above,  was  used  to  code  the  results  of  Experiment  2. 
The  Systems  category  consisted  of  14  codes  relevant  to  all  the  friendly  force  assets,  plus  toee 
addis  rial  categories  to  cover  miscellaneous  contingencies  in  the  data.  Chunks  could  receive 
mult)  ;  S  Systems  codes  if  multiple  assets  were  mentioned  in  a  single  chunk.  The  motivation  to 
include  a  Systems  code  was  to  provide  a  way  to  track  which  assets  were  discussed  most 
frequently,  with  the  potential  to  use  the  data  to  highlight  either  the  most  important  or  the  most 
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troublesome  assets.  In  practice,  however,  this  potential  was  not  fulfilled.  The  major  problem  for 
this  category  was  that  speakers  often  discussed  an  asset  without  mentioning  it  by  name.  The 
asset  being  referred  to  was  implicit  based  on  the  context  and  who  was  speaking  (as  certain 
positions  typically  controlled  particular  platforms).  In  many  cases  when  the  asset  was  hot 
explicitly  named,  it  could  be  inferred  from  the  context  by  the  coder;  however,  this  was  not 
always  the  case,  and  inferences  injected  increased  subjectivity  into  the  coding.  As  a  result,  the 
Systems  category  was  deemed  of  little  practical  value,  and  so  was  not  used  in  Experiments  3  or 
4. 

Valence 

A  perceived  shortcoming  of  the  verbal  coding  scheme  was  an  inability  to  distinguish 
diametrically  opposed  statements.  For  example,  the  statements  “I  can’t  get  it  to  move”  and  “He 
is  moving  on  plan  at  30  miles  per  hour”  would  receive  identical  codes  based  solely  on  Function 
and  METT-TC  categories.  While  these  statements  are  “about”  the  same  Move  function,  they 
convey  very  different  information  on  the  accomplishment  of  that  function.  Consequently,  a  new 
coding  category.  Valence,  was  introduced  and  used  in  Experiments  3  and  4  to  distinguish 
communications  conveying  positive  versus  negative  status  on  accomplishing  functions.  The 
Valence  code  was  always  given  in  relation  to  the  Function  code,  to  signify  whether  that  Function 
was  being  carried  out  successfully  or  not.  Each  verbal  chunk  was  scored  as  negative  (-1), 
neutral/  inconclusive  (0),  or  positive  (1)  with  respect  to  its  coded  function.  For  example,  “I  can’t 
get  it  to  move”  was  assigned  a  negative  value  (-1);  “Is  it  moving?”(without  a  verbal  response  to 
the  question)  was  assigned  a  neutral  value  (0);  and  “He  is  moving  on  plan  at  30  miles  per  hour” 
was  assigned  a  positive  value  (1).  The  valence  codes  did  not  address  the  tactical  goodness  or 
appropriateness  of  the  Function,  only  whether  the  Function  was  accomplished.  The  BDA 
Function  was  an  exeption;  however.  BDA  verbalizations  were  scored  as  negative  (-1),  if  (a) 

BDA  was  sought  but  no  useful  BDA  images  were  available,  or  (b)  images  were  available,  but  the 
images  indicated  that  the  enemy  asset  still  posed  a  threat.  BDA  verbalizations  were  rated 
positive  (1)  if  there  was  clear  information  Aat  the  target  had  been  disabled. 

Table  12  provides  examples  of  the  Valence  codes  assigned  to  verbal  chunks  from 
Experiment  3  for  each  C2  function,  and  Figure  6  illustrates  the  results  of  a  Function  by  Valence 
by  session-quartile  analysis  for  a  session  from  Experiment  3.  Overall,  by  far  the  majority  of 
verbal  communications  were  rated  positively  in  terms  of  Valence.  This  pattern  of  predominantly 
positive  status  on  accomplishing  C2  Fimctions  was  found  for  See,  Plan,  Move,  and  Strike  related 
communications.  In  contrast,  BDA  and  Other  related  communications  were  relatively  more 
negative.  For  BDA,  percentage  of  positive  versus  negative  verbalizations  was  nearly  equivalent. 
For  “Other”  commimications,  particularly  technical  status  comments,  communications  were 
predominantly  negative.  While  the  Other  communications  do  not  directly  relate  to  C2  Functions, 
they  provide  useful  information  on  player  perceived  technical  limitations  in  the  C2  prototype 
(e.g.,  slow  processing  and  system  crashes)  that  may  have  undermined  the  command  group’s 
performance  across  functions. 

Negatively  valenced  communications  provide  useful  diagnostic  information  on  the 
capabilities  of  the  C2  prototype.  Efforts  by  the  FCS  C2  program’s  Technical  Team  to  refine  the 
C2  prototype  should  attend  closely  to  the  problems  discussed  in  negatively  rated  verbalizations. 
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For  example.  Figure  6’s  quartiles  help  pinpoint  when  negative  verbalizations  occurred  in  the 
illustrated  session.  Dining  quartile  2,  player  verbalizations  indicated  problems  with  Move  and 
BDA  functions.  These  communications  stressed  the  difficulty  of  controlling  the  micro-UAVS, 
which  resulted  in  the  loss  of  two  of  them.  For  BDA,  negatively  valenced  communications 
indicated  the  participants’  inability  to  kill  designated  targets.  During  Quartile  4,  negative  Move 
related  comments  often  stressed  the  inability  to  maneuver  several  platforms,  including  Future 
Warriors,  Robo-Scout,  and  LOS. 

Valence  appems  to  be  a  useful  addition  to  the  coding  scheme.  Yet  it  should  be  noted  that 
it  requires  interpretation  on  the  part  of  the  coder  and  can  border  on  the  subjective,  depending  on 
the  chunk  content.  Note  that  no  attempt  has  been  made  to  assess  inter-coder  agreement  for  the 
Valence  code. 
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Figure  6.  Percent  of  verbalizations  by  function,  valence,  and  quartile  for  one  session  from 
experiment  3. 
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Table  12.  Examples  of  verbal  chunks  from  Experiment  3  by  Function  with  assigned  Valence 
values  of  positive  (1),  neutral/inconclusive  (0),  and  negative  (-1).  Note:  bullets  indicate  multiple 
speakers _  ^ 


Function 

Valence 

Examples  of  Verbal  Chunks 

Plan 

1 

•  I  was  just  thinking  about  the  birds  being  too  far  up  there,  up  North 

•  Bring  them  South  then. 

•  I  will,  I  don't  control  anything,  I've  got  to  ask  team  to  bring  them  further  South.  And  I  can 
do  that,  sir,  if  you  don't  mind. 

•  No  go  right  ahead,  keep  them  South,  that's  fine  with  me 

0 

•  I  got  an  idea,  do  you  want  to  try  something  new? 

•  No 

-1 

We  have  unconfirmed  as  of  yet  BDA  on  a  tank  m  the  South  and  a  couple  of  tanks  in  center  sector 
but  we  don't  have  enough  intelligence  yet  to  give  us  a  good  read  of  the  battlefield  other  than  the  fact 
that  he  tried  to  move  forward  in  the  center,  and  1  will  keep  you  informed 

1 

There’s  an  unknown  radio  hit. 

0 

They  haven’t  fired  any  artilleiy  yet,  have  they?  (no  response) 

-1 

Dang,  there  s  nothing  in  my  images  here. 

Move 

1 

•  So  we  need  to  get  those  2  micros  back  down  there 

•  They’re  coming  down. 

0 

Where  is  the  SAR  bird?  (no  response) 

-1 

That  one  s  stuck  there,  number  2  is  just  not.. .  microUAV  2  is  not  responding 

Strike 

1 

•  Did  you?  Did  you  fire  4? 

•  Yeah,  I  just  fired  4 

0 

Well,  the  question  is,  do  we  reengage?  (no  response) 

-1 

OK.  Interestingly,  you  lost  comms  on  the  PAMs  that  you  sent.  You  see  that?  PAM  54  lost  comms. 

It  didn  t  attack.  Hold  on  a  second,  the  last  2  PAMs  that  you  sent  lost  comms  and  did  not  go  to  the 
target.  You  want  me  to  show  you?  The  one  on  the  Daria  did  that  too.  Neither  one  hit  anything  They 
both  lost  comms.  j  ^  j 

BDA 

1 

Here  is  a  better  unage.  It  looks  like  it  might  be  perhaps  a  fire  power  kill,  maybe  a  fire  nower  - 

mobility  kill.  p  . 

0 

PAM  1 6,  where  did  that  hit?  (no  response) 

-1 

Is  it  broke?  Did  we  kill  it?’ 

I  don't  know,  it  doesn't  look  like  it's  broke  from  this  image  right  here,  it's  hard  to  tell 

Other 

1 

What’s  the  red  dot  mean?  - 

It  means  that’s  where  it  detected  something  and  takes  a  picture,  or  that's  the  place  where  the  Garm 
was  templated. 

0 

Blued,  Black  6  (no  response) 

-1 

I  ve  got  a  right  screen  frozen. 
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Command  Considerations 

Another  coding  category  used  for  Experiments  3  and  4  was  called  Command 
Considerations.  Adding  this  category  attempted  to  relate  participant  communications  with  key 
cognitive  patterns  important  to  battle  command.  The  nine  topics  used  to  assess  Command 
Considerations  are  listed  in  Table  13,  and  were  based  on  the  research  of  Lussier,  Shadrick,  and 
Prevou  (2003).  Coding  of  the  verbal  data  for  Command  Considerations  did  not  treat  the  nine 
codes  as  exclusive.  That  is,  the  same  verbal  chunk  could  be  assigned  multiple  command 
considerations,  because  as  can  be  seen  by  reading  Table  13,  the  considerations  are  not  mutually 
exclusive. 

Given  that  multiple  codes  could  be  assigned  to  one  chunk,  the  data  were  examined  by 
absolute  frequency,  rather  than  the  percentage  of  chunks  exhibiting  each  code.  We  expected  to 
find  evidence  for  the  occurrence  of  the  Command  Considerations,  given  the  background  and 
expertise  of  the  participants  in  the  experiments,  and  indeed  we  did  so,  as  illustrated  in  Figure  7. 
Example  verbal  data  for  each  Command  Consideration  is  given  in  Table  14. 


Table  13.  Command  considerations  for  analyzing  command  group  verbalizations 


Consideration 

Description  of  Consideration 

Plan 

Execution  is  self-initiated  and  preceded  by  plan  coordination/refinement. 

Inform 

Make  information  requirements  known. 

See 

Battlefield  visualizations  that  are  dynamic/predictive/proactive. 

Coordinate 

Create  synergistic  effects  with  multiple  assets/teamwork. 

Assets 

Use  all  assets  available. 

Situation  Awareness 

Continual  situation  assessment,  dynamic/contingency  planning. 

Terrain/Time 

Consider  effects  of  terrain/time. 

Enemy 

Model  a  thinking  enemy. 

Mission 

Keep  sight  of  the  big  picture  and  mission  intent. 

It  was  hypothesized  that  Command  Considerations  could  be  a  useful  measure  when 
comparing  teams  of  different  levels  of  expertise,  and  in  particular  as  a  measure  of  leader 
development  and/or  battle  command  expertise.  Gordon  (2001)  found  a  difference  in  the  number 
of  cognitive  statements  made  by  experienced  vs.  inexperienced  Air  Force  weapons  directors  in  a 
simulated  task  environment.  We  compared  the  behaviors  of  the  experts  who  participated  in 
Experiments  1-4  against  a  group  of  West  Point  Cadets  and  ROTC  students  who  participated  in  a 
separate  experiment  (between  Experiments  2  and  3).  As  the  novice  players  used  a  CSE  similar 
to  that  used  by  the  experts  for  Experiment  2,  the  data  from  one  expert  session  from  Experiment  2 
(deemed  most  similar  in  mission  to  that  given  the  novices)  was  analyzed  and  compared  to  the 
novice’s  data.  A  report  detailing  the  results  of  that  analysis  is  in  preparation  (Carnahan, 

Lickteig,  Sanders,  and  Durlach,  in  preparation).  Here  are  presented  just  the  results  for  Command 
Considerations  (see  Figure  8).  It  is  clear  from  inspection  of  Figure  8,  the  novices  produced  less 
frequent  evidence  for  Command  Considerations.  It  should  be  noted,  however,  that  the  single 
coder  who  assigned  Command  Considerations  was  not  blind  to  which  group  (experts  vs.  novices) 
was  being  coded.  Further,  no  attempt  was  made  to  assess  inter-coder  agreement.  The  coder 
(first  author)  who  assigned  the  codes  felt  the  assignments  were  very  subjective,  and  that  further 
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development  and  definition  of  the  code  elements  is  required.  Finally,  it  should  be  noted  that  for 
both  groups,  Command  Considerations  amounted  to  less  than  three  percent  of  verbalizations. 
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Figure  7.  Frequency  of  command  considerations  for  2  sessions  from  experiment  3.  Note: 
Command  considerations  defined  in  Table  13.  SA  =  situation  awareness. 
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Figure  8.  Percent  of  verbalizations  counted  as  a  command  consideration,  comparing 
observations  for  the  expert  vs.  the  novice  participants.  Note:  Command  considerations  defined 
in  Table  13. 
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Table  14.  Examples  of  Verbal  Communications  by  Command  Considerations  from  Experiment 
3 


Consideration 

Example 

Keep  sight  of  the  big 
picture  and  mission 
intent. 

I  think  they  are  probably  dead,  or  mobility  kills  which  for 
this  scenario  is  both  good. 

Yeah. 

We  won't  waste  any  more  rounds  on  him. 

Yeah.  Actually,  just  as  long  as  we  hit  him  even  if  they  are 
fire  power  kills,  I  don't  care  less. 

Jack,  what  we  don't  want  to  do  is  get  into  the  Netfires. 

Yeah  exactly,  exactly. 

So  that’s  why  I'll  leave  the  troop  transport  there,  to  protect 
our  flank. 

Protect  the  flank 

Model  a  thinking  enemy. 

That  means  he  is  moving  out  of  sector. 

I  mean  they  dropped  it  right  on  top  of  him. 

Either  that  or  he  is  trying  to  reposition  from  where  they  told 
us  he  was  at  Start  Ex. 

That  could  very  well  be. 

Right  in  the  middle  of  the  valley,  back  to  a  ridgeline,  or 
forward  into  the  gap  there. 

Consider  effects  of 
terrain/time. 

I'm  just  looking.  I'm  trying  to  find  that  freaking  keyhole  up  in 
here. 

Continual  situational 
assessment  and 
dynamic/contingency 
planning. 

I  stopped,  go  to  the  heads  up.  I  stopped  him,  that's  his  eyes 
so  if  this  guy  continues  in  this  direction  that's  a  keyhole. 

And  if  he  ends  up  here  I  ought  to  be  able  to  see  him  and  that 
would  be  within  Javelin  range. 

Okay. 

If  he  pops  up  in  any  of  these  areas  here. 

Use  all  assets  available. 

Brooks,  I  don't  think  we  are  going  to  get  the  A160  any  time 
soon,  so  I  need  you  “Sir”  to  get  the  GSR  up  on  some  higher 
groxmd  there  right  in  front  of  you  and  get  a  mast  up. 

Table  14  continued  on  next  page. 
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Table  14  continued. 

Create  synergistic  effects 
with  multiple 
assets/teamwork. 

Hey  Jack,  head's  up  chief,  here's  my  second  IFV  team  in  the 
South. 

Okay. 

It  runs... 

Now  that  tank. 

Yeah  we  got  to  kill  him.  He  dies  first. 

But  he  runs  here,  this  is  pretty  much  covering  the  meadow 
this  point  right  here  is  where  he  stops. .  .infantry  North. 

But  this  time,  all  hell  is  going  to  break  loose. 

All  of  a  sudden  two  different  infantry  teams  are  popping  up 
on  the  ridge,  well  one  on  the  ridge  and  one  on  the  road,  but 
they  are  only  300m  apart.  All  of  a  sudden  he's  got  clear 
visibility  across  the  valley. 

Very  nice,  now  which  one,  that's  the  IFV. 

flat's  the  IFV  with  2  mounted  teams. 

Okay. 

Now  I've  got  the  dismounts  starting  off,  2  routes,  this  guy  is 
here,  he  moves  here,  and  all  of  a  sudden. . . 

That's  a  pretty  good  view. 

He  gets  a  good  view  out  of  the  valley.  This  bobo  down  here. 

He  is  going  to  move  down  this  route,  and  he  should  be  able 
to,  shortly,  acquire  anything  out  in  that  open  area  there,  at 
this  position.  See  where  that  is  open? 

Oh  yeah. 

So  that's  right  now  his  route  plan. 

For  dismounts? 

Yeah. 

Battlefield  visualizations 
that  are  dynamic/ 
predictive/proactive. 

Hey  Dave,  if  I  was  a  guessing  man,  I  would  say  that  radio 
link  3,  which  is  up  there  where  that  PAM  lost  comms,  has 
now  become  unknown  27.  I'm  just  telling  you,  I  think  that 
very  well  may  have  moved  that  down  there. 

Make  information 
requirements  known. 

I  want  to  see  in  front  of  us  with  those  Micro-UAVs,  it  takes  a 
long  time  to  try  to  develop  what's  in  front  of  us,  so  we  need 
to  do  that  as  quickly  as  possible. 

Execution  is  self-initiated 
and  preceded  by  plan 
refinement/coordination. 

No  other  detections,  Micro-UAVs  going  out  there  taking 
pictures. 

Go  micro  go. 

Got  one  back  here.  Micro  1  waiting  for  its  detections  up 
there  in  the  North,  to  go  and  take  pictures  there.  The  other  3 
are  out  on  the  route. 
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Analysis  of  Planning  Phases 

As  mentioned  previously,  each  experimental  mission  was  preceded  by  a  planning  phase 
during  which  the  participants  planned  their  mission,  based  in  part  on  an  intelligence  report 
(which  was  different  for  each  session).  Observation  by  ARI  at  the  experimental  site  indicated 
that  key  C2  functions  were  taking  place  during  these  planning  sessions.  Consequently,  ARI 
arranged  to  capture  the  verbal  interactions  of  Experiments  3  and  4  during  the  planning  as  well  as 
the  mission  execution  phase  of  each  session.  Verbal  interactions  during  Experiments  3  and  4 
planning  phases  were  analyzed  in  the  same  manner  previously  described  for  the  mission 
execution  phases  of  Experiments  3  and  4.  Fidelity  of  the  verbal  recordings  was  lower  than 
during  execution,  however,  due  to  participants  neglecting  to  use  their  headsets  continuously 
during  planning  phases.  In  addition,  it  was  observed  that  some  planning  discussions  took  place 
outside,  as  well  as  within,  the  vehicle  simulator.  Consequently,  not  all  verbal  interactions 
related  to  planning  were  captured  for  analysis. 

Table  15  gives  some  of  the  characteristics  of  the  planning  phases  analyzed,  and  as  can  be 
seen  in  the  table,  planning  phases  were  typically  shorter  than  mission  execution  sessions,  with 
consequently  fewer  verbal  chunks.  The  percent  of  time  spent  verbalizing  during  planning  was 
less  than  during  mission  execution.  Possible  explanations  for  less  verbal  interaction  during 
planning  are  that  not  all  participants  remained  in  the  C^  vehicle  during  the  entire  planning  phase, 
and  their  headsets  were  not  continuously  worn  even  when  they  were  present.  It  was  also 
observed  that  participants  spent  more  time  working  individually  with  the  CSE,  entering  duty- 
specific  mission  and  planning  data  (e.g.,  routes  and  tasks)  into  their  computers. 


Table  15.  Key  characi 

:eristics  of  3  analyzed  planning  phases 

Session 

Experiment  3,  Run  10 

Experiment  3,  Runl  1 

Experiment  4,  Run.  8 

Session  Duration 

56  minutes 

39  minutes 

51.75  minutes 

Cumulative  Silence* 

7.6  minutes 

10.1  minutes 

7.5  minutes 

#  Verbal  Chunks 

75 

61 

62 

*  Timing  initiated  after  3  seconds  of  silence 


As  there  were  less  than  one  fourth  the  chunks  for  each  planning  phase  tiian  for  most 
mission  execution  sessions,  more  variability  in  the  relative  distributions  of  verbal  behavior  over 
the  codes  should  be  expected,  and  was  observed.  Mean  data  across  the  3  planning  phases, 
therefore,  will  be  presented,  and  compared  with  the  average  of  the  mission  execution  data 
presented  above.  Table  16  shows  these  data  for  the  Source  category.  As  during  execution, 
during  planning,  the  vast  majority  of  verbal  interactions  occurred  among  the  Black  cell  members. 
There  was  a  bit  more  communication  with  Blue  and  Other  (possibly  technicians),  and  less  with 
subordinates  (who  often  were  not  present  during  the  planning  phase). 

Figme  9  compares  planning  and  execution  verbalizations  categorized  according  to  the 
Type  of  interactions  that  occurred.  Whereas  Share  interactions  predominated  during  execution, 
in  planning,  it  was  Ask  interactions  that  were  most  prevalent.  The  relative  frequency  of  Action 
also  was  greater  than  during  execution.  These  data  comply  with  casual  observations  that  during 
planning,  the  participants  were  more  likely  to  report  to  one  another  what  they  were  doing  as  they 
were  doing  it,  and  to  check  with  one  another,  as  might  be  expected  during  planning. 
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Figure  10  compares  planning  and  execution  verbalizations  categorized  according 
to  Function  addressed  in  the  verbal  interactions.  Hardly  surprising,  during  planning,  Plan  was 
the  most  frequently  observed  Function.  Other  was  the  second  most  frequent  Function;  this 
reflects  the  participants’  tendency  to  talk  socially  and  less  formally  during  the  planning  phase. 

Table  16.  Mean  percent  of  verbal  interaction  observed  for  each  code  of  Source  category 


Source  code 

Planning 

Execution 

Within  Black  Cell 

78.3 

84.9 

Black  - 
Subordinates 

3.0 

7.2 

Black  -  Blue 

11.0 

4.5 

Black  -  White 

0.0 

0.3 

Blue  -  White 

0.0 

0.4 

3 -way 

0.0 

0.1 

Other 

7.3 

2.6 
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Figure  9.  Comparison  of  the  average  relative  distributions  of  the  type  category  for  planning  vs. 
execution  phases  for  sessions  from  experiments  2-4. 
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Figure  10.  Comparison  of  the  average  relative  distributions  of  the  Function  category  for 
planning  vs.  execution  phases. 

Figme  1 1  compares  planning  and  execution  verbalizations  categorized  according  to 
METT-TC.  Instead  of  being  almost  totally  concerned  with  Enemy  and  Troops,  as  during 
execution,  during  planning,  interactions  were  spread  more  evenly  among  the  7  METT-TC  codes. 
Although  Troops  still  predominated,  during  planning,  interactions  coded  as  Mission  were  the 
second  most  frequent,  followed  by  Enemy. 
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Figure  1 1 .  Comparison  of  the  average  relative  distributions  of  the  factor  category  for  planning 
vs.  execution  phases. 
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The  results  from  analysis  of  the  planning  phases  appear  to  differ  from  the  results  of 
analysis  of  the  mission  execution  sessions  of  these  experiments.  This  indicates  that  a  different 
pattern  of  interaction  occurred  during  planning  as  compared  with  execution.  It  also  indicates  that 
the  coding  scheme  was  able  to  detect  ^s  difference.  It  should  be  cautioned,  however,  that  the 
coder  was  not  blind  to  whether  a  session  undergoing  coding  was  from  planning  or  from 
execution. 


Wit^i  respect  to  C2  performance,  planning  behaviors  are  fundamental.  The  way  a  team 
performs  during  mission  planning  can  have  a  significant  effect  on  their  mission  situation  later.  A 
crew  that  conducts  thorough  planning  and  considers  alternative  contingencies  may  be  less  apt  to 
make  hasty  responses  or  to  lose  a  common  operating  picture  during  the  mission. 

Code  Combinations 


An  additional  approach  to  using  the  coding  scheme  could  be  to  examine  code  patterns 
across  categories.  By  examining  various  code  combinations,  it  would  be  possible  to  answer 
specific  questions,  should  they  arise.  For  example,  one  could  examine  the  Type  by  Source 
interaction  to  see  if  particular  Types  of  interactions  were  more  prevalent  among  certain  cells  in 
the  experiment  (see  Lickteig,  Sanders,  Durlach,  and  Carnahan,  2002  for  an  example). 

For  Experiments  2  (and  3)  we  examined  “code  profiles.”  That  is,  the  3-way 
combinations  of  Source-Type-Factor.  For  the  3  execution  sessions  analyzed  from  Experiment  2, 
198  Source-Type-Factor  profiles  were  used;  but  interestingly  13  profiles  accounted  for  50%  of 
the  data.  If  we  only  had  these  13  profiles,  we  still  could  have  coded  half  of  the  data  of 
Experiment  2.  All  of  these  13  profiles  were  coded  as  Within-Cell  for  Source.  The  other 
combinations  of  Type  and  Factor  are  listed  in  Table  17,  along  with  the  percent  of  the  data  each 


Table  17.  13  code  profiles  accounting  for  50%  of  the  chunks  of  the  data  analyzed  for  Experiment 
2.  All  these  were  interactions  am( 


Type 

Factor 

%  of  all 
chunks 

Share 

Locate  enemy 

7.7 

Share 

IT/CSE 

6.4 

Direction 

Strike-Lethal 

5.5 

Share 

BDA 

4.4 

Share 

Identify  Enemy 

3.7 

Process 

Enemy  Disposition 

3.1 

Share 

Mobility  Report 

3.1 

Action 

Strike-Lethd 

3.5 

Ask 

Strike-Lethal 

3.5 

Share 

Strike  Status 

2.9 

Share 

Sensor  Status 

2.7 

Ask 

Locate  Enemy 

2.4 

Share 

Strike-Lethal 

2.2 

Cumulative  Total 

51.1 
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profile  accounted  for.  One  could  consider  whether  some  of  the  most  frequent  profiles  could  be 
supported  by  automated  support  systems,  in  particular,  the  ones  sharing  new  information 
observed  from  the  CSE.  For  example,  Share-Locate  enemy  or  Identify  enemy:  the  CSE  could 
verbally  announce  relevant  information  when  new  enemy  icon  appears  on  the  situation 
awareness  map. 


Discussion  of  the  Coding  Scheme  Used  for  Experiments  2-4. 

The  coding  scheme  applied  to  the  verbal  data  of  Experiments  2-4  was  useful  in  providing 
quantitative  measures  describing  the  verbal  behavior  of  the  experimental  participants.  Moreover, 
the  scheme  was  sensitive  enough  to  reveal  different  patterns  of  verbal  interaction  during  mission 
execution  vs.  mission  planning,  and  different  patterns  for  a  novice  vs.  a  more  experienced  team. 
Despite  these  accomplishments,  there  are  several  ways  in  which  the  coding  approach  could  be 
improved. 

Firstly,  the  chunking  issue  was  never  fully  resolved.  Transcripts  were  chunked  first  by 
Type.  Then  on  a  second  pass  through  the  data,  any  chunks  for  which  multiple  other  codes 
seemed  to  apply  were  further  subdivided.  There  were  no  clear  rules  for  this  latter  subdivision; 
and  no  attempt  was  made  to  assess  whether  different  researchers  agreed  or  disagreed  >vith  the 
way  it  was  carried  out.  The  second  pass  through  infused  knowledge  of  the  coding  scheme  itself 
into  the  chunking,  which  ideally  would  be  completely  independent  of  the  coding  scheme 
subsequently  applied  to  it.  Breaking  the  text  down  into  smaller  chunks  made  it  easier  to  apply 
the  coding  scheme.  On  the  other  hand,  it  meant  that  certain  interactions  concerning  higher  level 
cognitive  processes  were  broken  down  into  several  chunks,  with  the  consequent  loss  of  the 
ability  to  capture  those  cognitive  processes  in  code  (see  Yang,  2003  for  further  discussion  of  this 
problem).  This  is  one  reason  the  new  category.  Command  Considerations  was  introduced  for 
Experiment  3.  However,  that  category  contains  codes  that  are  quite  vague  and  subjective,  and 
reworking  them  into  a  more  objective  instrument  would  be  usefol. 

For  the  analysis  of  Experiment  2  data,  efforts  were  made  to  establish  the  objectivity  of 
the  coding  scheme  by  assessing  inter-coder  agreement.  To  maintain  the  scientific  integrity  of  the 
data,  it  is  recommended  that  multiple  coders  and  the  assessment  of  their  agreement  always  be  an 
element  of  the  coding  process.  The  coding  is  a  cognitively  demanding  activity,  especially 
selecting  one  of  the  25  METT-TC  subfactors.  As  a  matter  of  human  nature,  the  coder  is  apt  to 
adopt  various  habits  or  biases  to  reduce  cognitive  load.  Awareness  that  code  assignments  will  be 
compared  to  those  of  another  coder  aids  in  preventing  criteria  shifting  during  coding. 
Measurement  of  inter-coder  agreement  also  provides  some  kind  of  index  regarding  the 
objectivity  of  the  coding  scheme  (Ericsson  and  Simon,  1993). 

It  is  anticipated  that  in  the  future,  the  verbal  coding  scheme  may  be  used  to  compare  the 
behavior  of  different  groups  of  participants,  or  the  same  participants  under  varying  conditions. 
Should  this  be  the  case,  it  is  recommended  that  coders  be  blind  to  the  nature  of  the  groups  or 
conditions  while  conducting  coding.  As  with  assessing  inter-coder  agreement,  blind  coding  will 
help  maintain  the  scientific  integrity  of  the  findings.  For  certain  types  of  comparisons,  this  may 
be  difficult,  despite  the  best  of  intentions,  however.  The  actual  content  of  the  transcripts  may 
provide  hints  as  to  the  nature  of  the  data  collection  conditions.  In  addition,  knowledge  of  the 
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context  in  which  the  data  were  produced  may  be  fundamental  to  understanding  transcript  content 
(see  Yang,  2003  for  a  discussion  of  this  point). 

The  coding  scheme  is  quite  complex  and  contains  several  redundancies.  In  particular,  the 
separate  categories  of  Function  and  Factor,  contain  substantial  overlap.  This  results  in  both  extra 
work  for  the  coder,  more  data  to  be  processed  and  presented,  and  more  complexity  for  the  end 
user  of  the  data.  Ideally,  Fimction  should  emerge  from  subsets  of  different  Factor  codes,  and  not 
be  a  separate  coding  category  in  and  of  itself  Below,  a  revised  coding  scheme  that  attempts  to 
do  this  is  presented. 

Neither  the  above,  nor  the  revised  scheme  presented  below,  are  hypothesis  driven.  The 
schemes  were  developed  largely  on  the  basis  of  doctrinal  tradition  (e.g.  l^TT-TC),  and  the 
results  generated  may  not  necessarily  be  significant  in  addressing  insightful  questions  about  the 
differences  between  operations  in  today’s  army  and  the  Furture  force.  Specific  hypothesis  driven 
analyses  may  require  further  code  refinement  or  the  addition  of  new  coding  categories. 

A  Revised  Coding  Approach  to  Factor 

As  a  result  of  the  short-comings  of  the  coding  scheme,  described  above,  an  attempt  was 
made  to  modify  the  scheme  subsequent  to  Experiment  4.  One  goal  of  the  revision  was  to 
streamline  the  scheme  to  make  it  easier  to  apply  and  present.  Focus  was  placed  on  the  Function, 
Factor,  and  Command  Considerations  categories.  It  is  recommended  that  application  and 
revision  of  other  coding  categories  should  be  governed  by  hypotheses  and  research  goals.  For 
example,  if  one  research  goal  were  to  investigate  patterns  of  communication  within  the  Unit  Cell, 
Source  could  be  modified  to  identify  specific  individuals  participating  in  the  interactions. 

The  Factor  category,  as  before,  was  based  on  METT-TC.  It  consisted  of  23  codes,  as 
listed  in  Table  1 8.  The  intention  was  to  compose  a  list  of  clearly  defined  topics  that  would  fit 
into  the  METT-TC  framework,  eliminate  codes  from  the  original  scheme  that  were  never  or  very 
rarely  used,  and  eliminate  redundancy  and  ambiguity.  Towards  the  goal  of  eliminating 
redundancy,  instead  of  coding  separately  for  Function,  as  before,  the  intention  was  to  derive 
Fimction  (and/or  Command  Considerations)  from  the  Factor  codes.  Our  proposed  method  of 
associating  Factor  codes  with  these  higher  order  categorizations  is  illustrated  in  Table  18. 

Another  goal  of  the  revision  was  to  capture  higher  level  cognitive  processes  within  the 
context  of  the  Factor  category.  Within  the  revised  scheme,  we  assert,  the  Plan  Function  captures 
the  intention  of  what  we  sought  with  Command  Considerations.  For  the  original  scheme,  the 
requirement  to  assign  one  and  only  one  code  to  each  chunk  of  data  meant  that  the  chunks  HaH  to 
be  relatively  short.  Consequently,  more  abstract  concepts  that  were  discussed  could  only  be 
perceived  by  reading  several  consecutive  chunks,  and  were  not  captured  by  the  Factor  category. 
For  the  revised  scheme,  we  changed  the  chunking  rules  and  allowed  coders  to  assign  as  many 
Factor  codes  to  a  chunk  as  they  deemed  appropriate.  Text  was  chunked  on  the  basis  of  natural 
communication.  That  is,  one  chunk  ended  and  another  began  when  there  was  a  natural  break  in 
the  conversation.  This  still  does  introduce  an  element  of  subjectivity,  in  that  a  coder  still  does 
need  to  decide  where  these  breaks  are.  However,  at  least  the  chunks  are  created  independent  of 
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Table  1 8.  Factor  Category  codes  for  the  revised  scheme  and  their  association  with  Function 


Code 

Factor 

Code  Definition 

Function 

1 

Mission 

Focus  on  mission  intent 

Plan 

2 

Mission 

Make  mission  critical  information  requirements  known 

Plan 

3 

Mission 

Make  responsibilities  known/delegation/role  clarification. 

Plan 

4 

Mission 

Maintain  common  operating  picture  (situation  report  or 
integration/summary  of  current  situation  or  interpretation  of 
situation;  create  common  understanding;  NOT  mere  citing  of 
enemy  targets  or  status  of  a  system) 

Plan 

5 

Mission 

Create  synergistic  effects  with  multiple  assets/teamwork  and/or 
use  all  assets  available 

Plan 

6 

Mission 

Create  or  modify  friendly  plan/COA,  based  on  current  intelligence 
(proactive) 

Plan 

7 

Mission 

Contingency  planning:  proactively  consider  events  that  may 
require  change  of  plan,  and/or  prepare  alternative  responses 
accordingly 

Plan 

8 

Mission 

Other  (concerning  Mission,  but  none  of  the  above,  include  mission 
rehearsal  here) 

Plan 

9 

Enemy 

Locate  enemy  positions,  movement,  or  identify  nature  of  enemy 
target 

See 

10 

Enemy 

BDA:  discuss  sensor  feedback  (including  eyes)  on  damage 
inflicted  on  enemy 

BDA 

11 

Enemy 

Model  a  thinking  enemy,  generate  probable  enemy  CO  A,  strategy, 
tactics 

Plan 

12 

Enemy 

Other  (concerning  enemy,  but  none  of  the  above,  e.g.,  if  enemy 
has  been  shot  at  and  by  who  or  what) 

See 

13 

Terrain 

Consider  effects  of  terrain 

Plan 

14 

Mobility  status  report 

Move 

15 

Troops 

Status  report:  any  other  status  report  or  assessment  of  friendly 
element  status  including  workload,  weapon  used  in  previous 
strike,  weapon  supply,  loss  of  asset,  communications,  or  other 
functionality 

See 

16 

Troops 

Information  processing  system:  functioning  of  the  CSE  and  use  of 
head’s  up  display 

See 

17 

Troops 

Immediate  Danger:  Awareness  of  immediate  dangers,  and  hasty 
reactions  taken  to  avoid  them,  such  as  a  survivability  move 

See 

18 

Troops 

Move:  deliberate  tasking/management/maneuvering  of  platforms 
(or  nonlethal  effects) 

Move 

19 

Troops 

Strike:  Launch/fire/deploy  lethal  or  potentially  lethal  effects 
(present  or  future),  including  order  to  fire  in  reaction  to  sighting 
enemy  target. 

Strike 

20 

Troops 

Other  (e.g.,  training,  anything  concerning  friendlies  that  is  not 
covered  in  8-18) 

See 

21 

Time 

Consider  time  constraints  or  timing  coordination 

Plan 

22 

Civilians 

Any  issues  concerning  civilians  /  noncombatants 

Plan 

23 

Other 

Other  (anything  else  not  covered  by  mission,  enemy,  terrain,  time, 
troops,  or  civilians,  can  include  joking  around,  other  non-mission 
related  interactions,  or  uninterpretable  interactions.) 

Other 
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the  coding  scheme  itself.  In  addition,  relaxation  of  the  constraint  of  one  and  only  one  code  per 
chunk  meant  that  all  content  could  be  coded  regardless  of  the  precise  method  of  chunking. 
Indeed,  it  could  be  argued  that  with  this  relaxation,  no  chunking  is  required  at  all.  The  codes 
could  be  applied  to  a  continuous  stream  of  verbal  interaction.  Nevertheless,  we  elected  to  use 
chunking.  Having  discrete  chunks  was  useful  for  purposes  of  data  analysis.  Without  some  way 
of  associating  codes  with  particular  sections  of  text,  there  would  be  no  way  to  accurately  assess 
inter-coder  agreement. 

One  reason  for  originally  allowing  one  and  only  one  code  per  chunk  was  that  it  followed 
the  dictates  of  classic  protocol  analysis  (Ericsson  and  Simon,  1993)  and  made  data  analysis 
relatively  straight  forward.  In  contrast,  if  coders  are  free  to  assign  as  many  codes  as  relevant  to 
each  chunk,  analysis  of  the  data  becomes  considerably  more  complex.  For  summarizing  such 
data,  one  approach  is  to  factor  in  the  total  number  of  codes  assigned.  Thus,  a  summary 
descriptive  statistic  for  each  code  element  could  be:  Percent  of  verbalizations  for  code  X  = 
Number  of  Verbalizations  coded  as  X  /  Number  of  codes  assigned  to  the  transcript.  Notice  that 
this  summary  statistic  is  independent  of  the  number  of  chunks  in  a  transcript.  Note,  however, 
that  chunking  does  still  have  consequences.  Under  the  revised  scheme,  a  chunk  that  has  repeated 
instances  of  die  same  subfactor  (e.g.,  multiple  sightings  of  the  enemy),  would  be  assigned  ^at 
code  only  once  (indicating  it  occurred  in  the  chunk,  but  not  how  frequently);  were  the  same  piece 
of  text  divided  into  multiple  chunks,  that  code  might  be  assigned  midtiple  times.  On  the  other 
hand,  this  will  partially  be  offset  by  calculating  the  percent  of  verbalizations  on  total  number  of 
codes  assigned. 

According  to  the  previous  scheme  (each  chunk  gets  one  and  only  one  code),  computation 
of  inter-coder  reliability  was  relatively  straight  forward;  however,  it  was  not  clear  what  to  do 
with  the  data  for  a  chunk  on  which  coders  disagreed.  Which  code  was  correct?  With  the  revised 
approach,  disagreement  among  coders  is  allowed,  and  in  a  sense,  adds  to  the  richness  of  the  data, 
for  it  allows  for  multiple  interpretations.  If  one  thinks  of  the  summary  percentages  as 
probabilities,  then  each  time  a  coder  assigns  a  particular  code,  it  raises  the  probability  that  a 
particular  topic  was  being  discussed.  If,  for  a  certain  piece  of  text,  all  coders  agree,  Ae 
probability  is  higher  than  if  each  assigns  a  different  code.  By  pooling  all  coders’  results,  and 
dividing  by  the  total  number  of  codes  assigned  (by  all  coders),  codes  for  which  there  is  more 
agreement  will  be  represented  as  higher  probabilities.  Codes  for  which  there  is  less  agreement 
will  not  be  discarded,  but  rather  will  be  represented  as  smaller  probabilities.  Thus,  high 
probabilities  imply  both  frequent  use  of  a  code  and  inter-coder  agreement.  Lower  probabilities 
are  ambiguous,  however;  they  could  imply  either  low  frequency  with  high  agreement,  or  high 
frequency  with  low  agreement  (that  is,  coders  use  the  code  often,  but  don’t  agree  on  the  chunks 
to  which  it  is  assigned).  Thus  it  is  necessary  to  introduce  some  measures  of  agreement. 

We  used  two  measures  of  agreement:  one  to  assess  inter-coder  agreement  (do  different 
coders  assign  the  same  codes  to  chunks?),  and  one  to  assess  code  consistency  (level  of  agreement 
among  coders  for  each  particular  code).  Both  of  these  computations  do  require  the  use  of 
chunks,  or  at  least  some  way  to  designate  which  piece  of  text  has  been  assigned  which  codes  by 
each  coder.  For  inter-coder  reliability  we  used  a  measure  that  varies  between  -1  and  1,  such  that 
1  indicates  perfect  agreement  among  coders  and  -1  complete  disagreement  among  coders.  0 
indicates  eqxial  amoimts  of  agreement  and  disagreement.  The  equation  used  to  compute  inter- 
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coder  agreement  is  given  in  Figure  12.  A  value  for  agreement  is  computed  for  each  chunk,  and 
then  the  scores  are  averaged  to  yield  a  mean  agreement  among  coders,  for  each  transcript. 


For  3  Independent  coders: 


Inter-coder 
AGREEMENT 
for  each  chunk 


(#  doubles  X  2)  +  (#  triples  x  3)  -  #  singles 
total  #  codes  assigned 


Where  #  triples  -  #  of  codes  used  by  all  3  coders 

#  doubles  =  #  of  codes  used  by  2  of  the  coders 

#  singles  =  #  of  codes  used  by  only  1  of  the  coders 


All  3  coders  assign 
a  code  of  7  to  a 
chunk. 


0 

All  3  coders  assign 
a  code  of  7  to  a 
chunk;  but  each  also 
assigns  another 
code  on  which  they 
disagree  (4,5, 6). 


All  3  coders  assign 
a  different  code  to 
the  same  chunk. 


Figure  12.  Explanation  and  illustration  of  the  measures  of  agreement  suggested  for  evaluating  the 
revised  coding  scheme.  Then  Venn  diagrams  depict  3  possible  coding  outcomes,  with  each  of  3 
coders  represented  by  one  of  the  circles  in  each  diagram. 

An  analogous  measure  was  computed  for  code  consistency.  Instead  of  computing  a 
measure  of  agreement  for  each  chunk,  a  value  is  computed  for  each  code,  by  examining  the 
agreement  concerning  which  chunks  the  code  was  applied  to.  This  measure  can  be  used  to 
interpret  the  relative  percentage  at  which  a  code  was  assigned,  which,  as  mentioned  above,  can 
be  ambiguous.  An  intermediate  percentage  with  a  high  code  consistency  implies  a  low 
frequency  with  consistent  assignment  of  the  code  across  coders,  whereas  the  same  percentage 
with  a  low  code  consistency  implies  a  high  frequency,  but  inconsistent  use  of  the  code  across 
coders. 


There  are  various  ways  in  which  inter-coder  agreement  and  code  consistency  could  be 
computed.  One  feature  to  note  regarding  the  measure  we  chose  is  that  it  penalizes  more  for 
disagreements  of  commission  (assigning  a  code  that  none  of  the  other  coders  used),  than  for 
omission  (not  assigning  a  code  that  the  other  coders  used).  In  particular,  a  value  of  1  can  be 
obtained  if  all  coders  assign  exactly  the  same  codes,  but  also,  if  there  is  a  discrepancy  only  of 
omission  (e.g.,  one  coder  assigns  codes  4,5,6,  another  assigns  4,5,  and  a  third  assigns  5,6). 


Application  of  Revised  Factor  Codes 


Two  graduate  students  from  the  University  of  Central  Florida  (report  co-authors)  served 
as  coders,  in  addition  to  the  first  author.  They  had  no  previous  expertise  in  the  FCS  domain, 
except  for  helping  to  transcribe  the  audio  data  from  some  of  these  experiments  into  written 
transcripts.  During  preliminary  meetings,  the  first  author  reviewed  the  features  and  terminology 
of  the  FCS  experiments,  and  a  revised  Factor  coding  scheme.  Randomly  selected  chunks  were 
examined  apd  the  codes  appropriate  for  each  were  discussed.  On  the  basis  of  these  discussions, 
the  code  definitions  were  further  refined  to  eliminate  ambiguity.  Independently,  each  coder  then 
coded  30  consecutive  chunks  of  text  from  Experiment  4;  and  the  coders  subsequently  met  once 
again  to  discuss  their  code  assignments,  and  further  questions  or  ambiguities.  This  process 
continued  until  all  coders  were  satisfied  that  we  had  reached  a  common  xmderstanding  of  the 
coding  scheme,  and  sufficiently  clear  code  definitions.  This  took  five  iterations.  Thereafter, 
coding  of  a  new  session  was  conducted  and  the  student  coders  were  advised  to  seek 
interpretation  from  the  firk  author  on  any  parts  of  transcripts  they  had  difficulty  understanding; 
the  interpretation  provided  explained  the  meaning  of  the  text,  without  reference  to  the  coding 
scheme  itself 

Analysis  of  Mission  Execution 

After  coder  training,  one  execution  session  from  Experiment  3  was  re-chunked  and 
independently  coded  by  the  three  coders.  The  new  chunking  approach  resulted  in  236  chunks 
(compared  with  461  from  the  original  scheme),  and  the  3  coders  applied  1407  codes  (an  average 
of  about  2  codes  per  chunk,  per  coder). 

Figure  13  illustrates  the  relative  frequency  of  the  METT-TC  factors  using  the  revised 
coding  scheme.  For  comparison,  the  results  for  Ae  same  session  using  the  original  coding 
scheme  are  also  shown.  As  can  be  see  in  the  figure,  the  relative  distributions  for  the  revised  and 
original  schemes  were  essentially  identical.  One  difference  was  a  relatively  higher  proportion  of 
Mission  factor  verbalizations  coded  under  the  revised  scheme. 

Figure  14  illustrates  the  Functions  derived  from  the  revised  Factor  coding.  For 
comparison,  the  results  for  the  same  session  using  the  original  coding  scheme  are  also  shown. 

As  can  be  seen  in  the  figure.  See  was  the  most  frequently  observed  function  for  both  schemes; 
however,  the  difference  between  See  and  the  next  most  frequently  observed  function  was  greater 
for  the  revised  scheme  (53.9  vs.  1 7.4)  than  for  the  original  scheme  (3 1 .7  vs.  23 .9).  Moreover,  in 
the  revised  scheme,  the  second  most  frequently  observed  function  was  Plan,  as  opposed  to  the 
original  scheme,  for  which  both  Strike  and  Move  were  relatively  more  frequent  than  Plan.  This 
most  likely  reflects  the  revision  of  the  coding  scheme  to  better  capture  command  considerations 
and  other  aspects  of  planning  that  were  difficult  to  capture  in  the  original  scheme. 

Figures  15, 16,  and  17  illustrate  how  the  METT-TC  codes  were  distributed  across 
specific  subfactors.  Subfactors  contributing  to  the  function  Plan  are  illustrated  in  Figure  15. 
Those  associated  with  the  function  See  are  illustrated  in  Figure  16,  and  the  remaining  subfactors 
(associated  with  Move,  Strike,  and  BDA)  are  illustrated  in  Figure  17.  The  number  in  parentheses 
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N/ission  Enemy  Terrain  Troops  Time  Civilians  Other 


□  Revised 
■  Original 


Factor 


Figure  13.  Relative  frequency  of  the  METT-TC  factors  for  an  execution  session,  for  the  revised 
vs.  the  original  factor  schemes. 


□  Revised 
■  Original 


Plan  Move  See  Strike  BDA  Other 


Function 


Figure  14.  Relative  frequency  of  the  function  categories  for  an  execution  session,  for  the  revised 
vs.  the  original  coding  schemes. 

above  each  bar  in  these  figures  represents  code  consistency  (for  each  code,  the  degree  to  which 
coders  agreed  in  its  use).  These  numbers  are  also  shown  in  Table  19,  along  with  the  number  of 
observations  for  each  code  (the  more  observations  made  for  each  code,  the  more  confidence  one 
can  have  that  the  measure  is  meaningful). 
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Figure  15  shows  that  Delegation  was  the  most  frequently  observed  Plan  subfactor,  and 
that  it  had  a  relatively  high  code  consistency  of  .42.  On  the  other  hand,  except  for  Civilians,  the 
code  consistency  of  the  other  subfactors  was  rather  lower  than  we  would  have  liked.  These 
subfactors  were  the  most  abstract  and  therefore  required  the  most  interpretation  on  the  part  of  tlie 
coders.  Thus,  achieving  code  use  consistency  for  these  may  be  a  perpetual  challenge.  However, 
for  some  cases,  we  feel  better  agreement  could  be  achieved  with  more  coding  training  and 
practice.  For  example,  the  two  student  coders  were  biased  against  using  Time  or  Terrain  (this 
came  out  in  subsequent  discussion).  For  other  cases,  the  codes  were  used  so  rarely  that  the 
measure  derived  for  code  consistency  is  probably  not  very  meaningful  (e.g..  Mission  Focus  was 
only  used  one  time  by  one  coder,  resulting  in  a  consistency  score  of  -1). 

Figure  16  shows  the  subfactors  associated  with  function  See.  For  these  subfactors,  code 
consistency  was  relatively  good,  ranging  from  .34  to  .70.  The  most  frequent  of  the  subfactors 
associated  with  See  were  status  reports,  followed  by  enemy  locations/identifications.  Figure  17 
shows  the  subfactors  associated  with  Move  (Mobility  and  Move  Action),  Strike,  and  BDA.  As 
for  the  See  subfactors,  code  consistency  for  these  was  fairly  good,  ranging  from  .52  to  .71. 

Figure  18  illustrates  the  distribution  of  scores  for  inter-coder  agreement  across  chunks. 
For  each  chunk,  a  measure  of  inter-coder  agreement  was  computed  as  described  in  Figure  12.  As 
can  be  seen  in  Figure  18,  the  coders  agreed  more  than  they  disagreed  (agreement  >0)  about  90% 
of  the  time. 
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Figure  15.  Relative  frequency  of  the  subfactors  associated  with  the  function  Plan,  when  the 
revised  coding  scheme  was  applied  to  a  mission  execution  session. 
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Figure  17.  Relative  frequency  of  the  subfactors  not  associated  with  plan  or  see,  when  revised 
coding  scheme  was  applied  to  an  execution  session. 
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Figure  18.  Distribution  of  inter-coder  agreement  scores  when  the  revised  scheme  was  applied  to 
an  execution  session.  Each  of  the  236  chunks  was  assigned  an  inter-coder  agreement  score, 
scores  could  range  from  -1.0  (complete  disagreement)  to  1.0  (complete  agreement). 

Analysis  of  Planning  Phase 

To  further  assess  the  appropriateness  of  the  revised  coding  approach,  we  also  applied  it  to 
one  planning  session  from  Experiment  4.  Rechunking  produced  63  chunks.  237  codes  were 
assigned  by  the  three  coders,  for  an  average  of  about  1.5  codes  per  chunk,  per  coder.  Figure  19 
illustrates  the  obtained  relative  frequency  of  the  METT-TC  factors,  as  well  as  the  results  for  the 
same  session  using  the  original  coding  scheme,  for  sake  of  comparison.  As  can  be  see  in  the 
figure,  the  relative  distributions  for  the  revised  and  original  schemes  were  similar,  except  that 
under  the  revised  scheme.  Troops  was  relatively  less  frequent  and  Other  was  relatively  more 
frequent,  than  for  the  original  scheme.  This  was  largely  due  to  the  fact  that  many  verbalizations 
that  had  been  coded  as  Troops-Other  (code  20,  Table  4)  under  the  original  scheme,  were  coded 
as  Other  (code  23,  Table  18)  under  the  revised  scheme. 

Figure  20  illustrates  the  same  data,  but  now  categorized  by  Function.  The  results  were 
similar  for  the  original  and  revised  coding  schemes,  except  for  the  relative  frequencies  of  See 
and  Strike,  with  See  being  more  frequently  coded,  and  Strike  less  frequently  coded,  under  the 
revised  as  compared  with  the  original  scheme.  For  those  relatively  frequently  coded  functions. 
Plan  and  See,  the  results  are  further  broken  down  by  code  in  Figures  21  (Plan)  and  22  (See). 

For  Plan  (Figure  21),  Mission-Other  was  the  relatively  most  frequently  coded  topic.  The 
subfactor  included  mission  rehearsal,  as  well  as  other  interactions  that  were  mission-relevant,  but 
not  well  captured  by  the  other  subfactors.  Code  consistency  was  generally  better  for  these 
subfactors  for  the  plaiming  session,  than  for  the  execution  session  (see  Table  19  for  a  direct 
comparison);  despite  the  fact  that  there  were  less  data.  This  may  be  because  planning 
interactions  during  planning  sessions  may  be  more  obviously  about  planning  (e.g.,  due  to  use  of 
future  tense,  or  explicit  discussion  of  “the  plan”),  than  such  interactions  during  execution.  It  may 
also  be  due  to  the  coders’  knowledge  that  tihey  were  coding  a  planning  session;  having  that 
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knowledge  they  may  have  been  more  prone  to  use  planning-associated  codes  (however,  coders 
were  not  explicitly  aware  of  which  codes  contributed  to  function  Plan  while  coding,  as  this  was 
derived  later  and  not  represented  on  the  coding  plan). 
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Figure  19.  Relative  frequency  of  the  METT-TC  factors  for  a  planning  session,  for  the  revised  vs. 
the  original  factor  schemes. 

For  See  (Figure  22),  Locate  enemy  was  relatively  more  frequent  than  any  of  the  other 
associated  codes,  and  had  a  fairly  good  level  of  consistency  (.68).  The  other  codes  associated 
with  See  had  a  poor  level  of  consistency  for  planning  (less  than  0). 

Figure  23  illustrates  the  distribution  of  inter-coder  agreement  scores  across  the  63 
chunks.  Agreement  was  fairly  good,  with  a  mean  score  of  0.41,  despite  the  fact  that  there  were 
relatively  few  chunks.  Eighty-seven  percent  of  the  scores  were  0  or  greater. 

Table  19  lists  code  consistency  for  the  execution  and  planning  sessions,  in  order  to  allow 
an  examination  of  how  these  varied  from  one  session  to  another.  In  actuality,  there  is  probably 
not  enough  data  here  to  determine  whether  certain  codes  are  used  more  consistently  than  others. 
Further  refinement  of  the  coding  scheme  could  entail  application  of  it  to  more  data,  with  an 
attempt  to  assess  and  further  revise  codes  that  coders  seem  unable  to  use  consistently.  Of  course, 
other  forms  of  assessment  are  also  possible,  for  example  examination  of  intra-coder  reliability 
(same  coder  coding  the  same  transcript  multiple  times). 
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Table  19.  Code  consistency  scores  associated  with  each  of  the  23  Factor  subcodes  of  the  revised 
coding  scheme,  for  an  execution  session,  and  a  planning  session.  Numbers  in  parentheses  give 
the  actual  number  of  times  that  code  was  assigned  by  the  coders _ _ 


■ 

Factor 

Code  shorthand 

Function 

Consistency 

Execution 

(^observations) 

Consistency 

Planning 

(#observations) 

1 

Mission  Focus 

Plan 

-1.0(1) 

IHHDSBIHHI 

2 

Mission 

Critical  Information 

Plan 

■HlEnSEI 

3 

Mission 

Delegation 

Plan 

■HKEISUH 

4 

Mission 

Common  Operating  Picture 

Plan 

0.21  (43) 

5 

Mission 

Teamwork 

Plan 

6 

Mission 

Course  of  Action 

Plan 

0.25  (16) 

7 

Contingencies 

■EfEHEnHi 

8 

Mission 

Mission  Other 

Plan 

■KiiRESlH 

9 

See 

BDA 

BDA 

HKEmHH 

11 

Plan 

HHOnESH 

12 

Enemy  Other 

See 

HIHEISSHH 

13 

Terrain 

Terrain 

Plan 

HIEElEEHHi 

14 

lUGSSJsSH 

Mobility 

Move 

■KiEEm 

15 

EES539I 

Status 

See 

16 

BSSSi 

CSE 

See 

17 

BSSSi 

Danger 

See 

HHBGSHHI 

18 

ISSBIIHlii 

Move  Action 

Move 

0.54(119) 

19 

Troops 

Strike 

Strike 

■EEHESH 

20 

Troops 

Miscellaneous 

See 

21 

Time 

Time 

Plan 

22 

Civilians 

Civilians 

Plan 

23 

Other 

Other 

Other 

.17(29) 

0.87  (75) 

A  comparison  of  Figures  14  and  20  reveals  that  the  revised  scheme  produces  different 
relative  distributions  over  Function  for  planning  and  mission  execution  phases.  Likewise,  a 
comparison  of  Figures  13  and  19,  shows  this  is  also  true  for  METT-TC  Factor.  Thus,  the  revised 
scheme  still  seems  to  discriminate  the  two  phases. 

Conclusions 

The  present  report  described  the  evolution  of  a  coding  scheme  for  verbal  interactions  in 
an  FCS  C2  setting.  The  work  was  exploratory  in  that  coding  was  intended  to  characterize  the 
verbal  interactions  observed,  but  was  not  aimed  at  testing  any  particular  hypothesis.  We  suggest 
that  the  revised  coding  scheme  presented  and  illustrated  in  the  latter  sections  is  of  potential  use 
for  future  hypothesis-driven  research  in  the  FCS  C2  arena.  The  METT-TC  Factor  coding 
scheme  could  be  used  in  conjunction  with  other  coding  categories  prescribed  by  the  research 
question,  to  provide  a  quantitative  analysis  of  verbal  interactions.  The  advantages  of  the  revised 
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Figure  20.  Relative  frequency  of  the  function  categories  for  a  planning  session,  for  the  revised 
vs.  the  original  coding  schemes. 


Factor  coding  scheme  are  that  it  allows  room  for  interpretation  (coders  are  allowed  to  disagree), 
and  captures  concrete  as  well  as  more  abstract  conversational  content  relevant  to  C2.  The  former 
could  also  be  seen  as  a  disadvantage,  however;  as  it  is  not  clear  how  much  disagreement  should 
be  acceptable  before  the  scheme  is  deemed  unreliable.  Only  for  a  large  number  of  respondents 
(coders),  are  there  techniques  (multivariate)  for  handling  such  uncertainty.  Another 
disadvantage  is  that  the  data  analysis  is  somewhat  more  complex,  compared  with  the  one  chunk 
-  one  code  approach. 

The  effort  and  time  involved  in  analyzing  the  data,  on  the  other  hand,  pales  in 
comparison  to  the  effort  and  time  involved  in  transcribing  the  audio  recordings  of  the  verbal 
interactions  into  written  transcripts.  This  task  is  required  for  any  coding  scheme  and  should 
always  be  taken  into  consideration  when  deciding  whether  to  conduct  a  verbal  analysis  to  begin 
with.  The  second  most  intensive  task  is  training  the  coders.  This  is  particularly  the  case  when 
coders  have  little  familiarity  with  the  content  to  be  coded,  and  need  to  learn  subject  matter  and 
specialized  terminology,  ^^en  this  is  the  case,  it  is  hard  to  determine  if  inconsistent  usage  of 
codes  across  coders  is  due  to  differences  in  interpretation  or  merely  lack  of  understanding.  In  the 
ideal,  the  coding  scheme  would  be  applied  by  subject  matter  experts.  Despite  efforts  to  be 
“objective,”  analysis  of  verbal  data  such  as  that  discussed  here  always  has  an  element  of 
interpretation  (Yang,  2003).  In  particular,  coding  of  the  more  abstract  verbal  elements  (e.g., 
model  a  thinking  enemy)  was  less  consistent,  than  coding  of  more  concrete  verbal  elements  (e.g., 
command  to  strike).  This  may  be  a  perpetual  problem  even  with  extensive  training  of  coders. 

We  suggest  that,  beyond  refining  the  coding  scheme,  and  training  coders,  the  only  way  to  get  a 
handle  on  this  “subjectivity”  is  to  measure  it,  and  interpret  the  data  in  the  light  of  the 
information.  If  the  code  is  applied  to  experimental  data  (i.e.,  data  collected  under  systematically 
varying  conditions),  it  is  essential  that  coders  be  blind  to  those  conditions,  for  the  results  to  be  as 
objective  as  possible. 
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Figure  21 .  Relative  frequency  of  the  subfactors  associated  with  the  function  plan,  when  the 
revised  coding  scheme  was  applied  to  a  planning  session. 
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Figure  22.  Relative  frequency  of  the  subfactors  associated  with  the  function  see,  when  the 
revised  coding  scheme  was  applied  to  a  planning  session. 


Figure  23.  Distribution  of  inter-coder  agreement  scores  when  the  revised  scheme  was  applied  to 
a  planning  session.  Each  of  the  63  chunks  was  assigned  an  inter-coder  agreement  score.  Scores 
could  range  from  -1.0  (complete  disagreement)  to  1.0  (complete  agreement). 
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ACRONYMS 


AGM 

Attack  Guidance  Matrix 

ARI 

Army  Research  Institute 

BDA 

Battle  Damage  Assessment 

C2 

Command  and  Control 

C4ISR 

Command,  Control,  Communications,  Computers,  Intelligence, 
Surveillance,  and  Reconnaissance 

CECOM 

Communications-Electronics  Command 

comms 

communications 

CSE 

Commander’s  Support  Environment 

DARPA 

Defense  Advanced  Research  Projects  Agency 

PCS 

Puture  Combat  System  (of  systems) 

PLOT 

Porward  Line  of  Troops 

PWC 

Puture  Warrior  Carrier 

IPV 

Infantry  Field  Vehicle  (same  as  PWC) 

LAM 

Loitering  Aerial  Munitioin 

LOS 

Line  of  Sight 

MTI 

Moving  Track  Indicator 

mUAV 

Micro  UAV 

NLOS 

Non-Line  of  Sight 

OTB 

One-SAF  Testbed 

PAM 

Precision  Aerial  Munition 

RDEC 

Research  and  Development  Center 

ROIC 

Reserve  Officer  Training  Corp 

SAP 

Semi-Automated  Forces 

SAR 

Synthetic  Aperture  Radar 

STO 

Science  and  Technology  Objective 

UAV 

Unmaimed  Aerial  Vehicle 

UGS 

Unattended  Groimd  Sensor 
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SAMPLE  UNCHUNKED  AND  UNCODED  TRANSCRIPT,  EXPERIMENT  4,  SESSION  8, 

03/03. 


Commander 

Let  us  turn  on  the  AGM  please.  Execute  the  AGM  please. 

Effects 

AGM  executed. 

Commander 

Roger 

Commander 

And  there  goes  the  AGM. 

Effects 

Beautiful! 

Commander 

1  did  not  see  the  SA  in  the  south.  1  guess  he's  targeted,  the  SA-15. 

Battlespace  Manager 

He's  been  targeted. 

Commander 

And  Roger 

Effects 

Hold  on.  1  have  threat  manager  up? 

Effects. 

SA15,  MPERM 

MPERM 

Battlespace  Manager 

It's  been  fired. 

Commander 

Right,  and  Roger 

Should  be  waiting  for  our  chaser. 

Commander 

And,  Roger. 

There's  the  chaser. 

Commander 

Now  look  Ken,  I  want  you  to  ask  higher  to  help  you  in  the  south  and  you 
concentrate  on  the  north. 

Commander 

Higher's  pulling  up  those  pictures? 

Intel  Manager 

Higher's  working  on  the  south  and  deep. 

Commander 

Alright  gentlemen, . 

Intel  Manager 

And  I'm  focused  on  the  center. 

Commander 

Good. 

Intel  Manager 

I'm  going  to  focus  on  the  route. 

Commander 

SA-15  smoking,  change  its  state  please,  gentlemen. 

Effects  . 

It's  a  beautiful  thing. 

Commander 

Oh,  hey,  you've  got  something  smoking  next  to  it  too.  That  chaser,  head's  up. 

Intel  Commander 

Yes 

Commander 

Watch  this.  The  chaser  that's  just  north  of  it,  looks  like  a  smoking  Dreaga,  not 
a  scout  team.  Am  I  wrong? 

Intel  Commander 

What  chaser  is  that? 

Commander 

Chaser  2,  not  chaser  1 . 

Battlespace  Manager 

Radio  unknown  6. 

Commander 

It  looks  to  me  like  a  Dreaga,  gentlemen.  What  do  you  say  Ken?  You're  the 
man. 

Intel  Manager 

1  would  say  that's  a  Dreaga.  1  would  say  that's  a  smoking  Dreaga. 

Commander 

1  think  it's  a  smoking  Dreaga. 

Commander 

I'll  change  it.  I'll  change  it. 

Intel  Manager 

And  a  URALs  truck  is  smoking  too.  Chaser  4. 

Effects 

Truck  killer,  truck  killer. 

Battlespace  Manager 

You  gonna  change  that,  Ken? 

Intel  Manager 

I'll  change  it. 

Battlespace  Manager 

Ok,  you've  got  a  chaser  7. 

Intel  Manager 

1  changed  it. 

Battlespace  Manager 

Looking  at  chaser  7.  Yes  baby,  yes. 

Battlespace  Manager 

Looky  here  on  the  heads  up. 
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Commander 

Quick  fires,  net  fires  launcher. 

Battlespace  Manager 

How's  that  for  chaser,  Jack? 

Commander 

Oh  very  nice.  Change  state  please.  We'll  take  that  one. 

Effects 

It  appears  to  be  a  Dreaga. 

Commander 

It  appears  to  be  a  Dreaga.  Ok  this  tells  me  he  is  defending  forward  along  the 
PLOT.  And  now  which  chaser  was  that,  sir? 

Battlespace  Manager 

That  was  chaser  7. 

Commander 

Tell  me  it  was  7,  it  was  7? 

Battlespace  Manager 

Yes 

Commander 

That's  all  1  wanted  to  know,  ok,  good,  1  got  it. 

Commander 

And  Blue6-Black6. 

Blue 

Blue6.0ver. 

Commander 

Alright  Blue6,  Blacks.  Initial  read  of  the  battlefield  is  that  he  is  defending 
fonward  with  a  heavy  security  zone  established  along  the  PLOT.  Break. 

Commander 

1  detect  at  least  2  platoons  forward  along  the  PLOT  supported  by  air  defense 
artillery.  Break. 

Commander 

Additionally,  it  looks  like  he's  also  got  some  concentration  in  the  center. 

Commander 

Therefore  1  suspect  a  heavy  forward  defense,  a  heavy  security  zone  with  a 
mobile  defense  more  than  likely  coming  from  the  center  sector.  Break. 

Commander 

Therefore  1  think  I've  got  all  the  information  1  need  to  begin  executing  decision 
point  1  and  our  attack  in  the  north.  Over. 

Blue 

Roger,  1  think  you'll  find  ...  by  Granite  Pass,  on  the  back  side  of  Iron 

Triangle.  1  think  we  have  a  good  read.  Over. 

Commander 

Ok,  what  1  need  now,  is  1  need  you  to  switch...  have  you  painted  everything 
with  the  A160?  Over. 

Blue 

Roger,  we  are  in  the  process  of  covering  the  box.  Over. 

Commander 

Ok  we  need  to  switch  in  just  a  minute  and  go  SAR.  Over. 

Intel  Manager 

Yeah,  if  they  can  doT1... 

Commander 

In  fact  if  you  can  do  T1 ,  we  are  ready  for  that  now.  Over. 

Commander 

Blue-Black,  did  you  copy  last? 

Blue 

I'm  going  to  check  that  now. 

Commander 

Thank  you  sir. 

Commander 

Oh  Ken, 

Intel  Manager 

Yes,  sir 

Commander 

I  need  you  to  keep  an  eye  on  what  higher's  doing  and  how  they  are  doing  it 
for  us. 

Intel  Manager 

Will  do. 

Commander 

And  David,  we  have  moving  tracks  beyond  Marjorie,  2  of  them,  that  would  be 
suspected  track  801  and  suspected  track  03.  Go  ahead  and  engage  those 
please. 

Blue 

This  is  blue.  SAR  bird's  inbound,  will  take  8  minutes  to  complere.  Over 

Commander 

And  Roger,  Thank  you  very  much. 

Commander 

Ah,  David,  on  order  I  want  you  to  launch  Lams  north  into  along  the  route  to 
protect  Brooks. 

Effects 

I  already  have  one  out  there. 

Commander 

Yeah,  your  center  with  that,  I  didn't  see  one  in  the  north. 

Effects 

Well  we  got... 

Ahhhh... 

Effects 

He's  mine. 
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Commander 

He's  yours. 

Commint. 

Commander 

Double  tap,  1  want  him  double  tapped.  Brooks,  you  take  him  too. 

Intel  Manager 

And  I've  got  a  UAV  that  is  just  about  to  go  there. 

Commander 

Excellent,  excellent,  good  coordination.  All  three  battle  stations  working 
together  to  kill  that  critical  target  on  the  battlefield  and  make  our  maneuver  a 
success. 

Effects 

You  sound  like  a  damn  commercial. 

Commander 

You  fiked  that,  huh? 

Intel  Manager 

We  cleared  that  area  out. 

Effects 

Ahhhh... 

Commander 

Famous  last  words,  there  may  be  some  tanks  there  we  can't  find.  Anyhow. 

Commander 

What  was  that  in  the  south  there,  Dave? 

Effects 

I'm  having  some  iconage  problem. 

Commander 

Alright. 

Commander 

Your  micro  went  past  him  there. 

Intel  Mariager 

Hey  you  know  what? 

Commander 

What? 

Intel  Manager 

I've  got  the  other  one  going  towards  him  because... 

Commander 

Ok,  you  do  whatever  you  have  to  do. 

Intel  Manager 

If  1  went  in  there  too  early,  he's  got  a  chance  of  getting  knocked  down. 

Commander 

I’m  with  you.  You  do  what  you  gotta  do.  I'm  not  telling  you  how  to  do  it.  1  am 
just  telling  you  what  1  want. 

Intel  Manager 

Got  it  sir. 

Commander 

Hooah 

Commander 

Ah,  it  looks  like  all  the  Ugs  are  in.  That's  good. 

Battlespace  Manager 

Unknown  9,  center  sector,  MTI  SAR. 

Effects 

Let  me  check  him  out  on  unit  viewer  real  quick.  Is  he  moving? 

Intel  Manager 

That  chaser  round  ... 

Commander 

Yeah,  it's  ok.  It's  alright. 

Commander 

Due  to  the  PAM  hit. 

Intel  Manager 

Roger 

Commander 

Whenever  it  hits.  It's  inbound. 

Commander 

Alright  Brooks,  you  take  that  un...Ah,  1  don't  know  man,  that  unknown  is  bad. 

Battlespace  Manager 

It's  alright. 

Commander 

No,  no,  1  mean  1  don't  know,  you  know,  if  he's  unknown,  yeah  that's  what  I'm 
talking  about. 

Battlespace  Manager 

Another  radio,  radio  12. 

Effects 

Where? 

Battlespace  Manager 

I  got  him. 

Effects 

I  can  get  him  too. 

Battlespace  Manager 

Let's  put  a  cargo  out  there  and  see  what  we  got. 

Battlespace  Manager 

Another  radio  in  scotch,  you  got  that  one.  Unknown  13. 

Effects 

1  got  him. 

Battlespace  Manager 

Got  him? 

Commander 

OK,  he  seems  to  be  moving  north  with  some  elements. 

Effects 

Unknown  9  bothers  me. 

Commander 

Unknown  9  is... 

Effects 

Unknown  mover. 
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Commander 

Unknown  13  is  beyond  Leslie. 

Effects 

That's  why  it's  bothersome. 

Battlespace  Manager 

It's  not  moving.  It  was  seen  by  SAR.  SAR  sweep  got  it. 

Commander 

Got  which  one? 

Battlespace  Manager 

We  got  100%  confidence  and  its  unknown. 

Intel  Manager 

Center  Sector  there. 

Commander 

It's  found  by  MTI  though,  it's  moving.  It's  found  by  MTI. 

Battlespace  Manager 

It  was  found  by  MTI  SAR 

Commander 

Continue  reading  the  sentence.  It  says  using  MTI. 

Battlespace  Manager 

Probably  a  LAV. 

Commander 

Yeah,  that's  what  I’m  thinking. 

Battlespace  Manager 

I’ll  put  a  tag  on  him  temporarily. 

Commander 

Ok  question,  urn,  Dreaga  chaser  7,  Dreaga  chaser  1 .  Are  there  2  dreagas 
there  or  are  there  one  Dreaga  there? 

Battlespace  Manager 

Where  you  talking? 

Commander 

Ken? 

Intel  Manager 

Is  that  it  on  the  PLOT? 

Commander 

Yeah 

Intel  Manat 

1  think  there  is  one  Dreaga  there  that.l... 

Commander 

1  want  a  micro  to  check  that,  because  in  that  position,  he  can  kill  Brooks 
coming  out  of  the  pass.  1  need  to  verify  that  that’s  dead. 

Intel  Manager 

He  was  only  fired  upon  one  time. 

Commander 

Ok  Dave,  1  might  need  you  to  double  tap. 

Effects 

On? 

Commander 

I’ve  got  2  Dreagas  there.  I’ve  got  7  and  1. 

Effects 

Where? 

Commander 

Center  sector,  PLOT  south  of  north. 

Battlespace  Manager 

I'm  looking  at  14. 

Effects 

You’re  confusing  me. 

Commander 

Heads  UP 

Effects 

Yeah 

Commander 

Right  here.  1  am  worried  that  there’s  2  tanks  there,  not  one.  OK? 

Intel  Manager 

Alright. 

Commander 

Or  maybe  we  killed  a  decoy. 

Effects 

Where? 

Battlespace  Manager 

I’m  thinking,  go  to  heads  up. 

Effects 

Show  me. 

Commander 

Dave,  reengage  that  for  me  buddy. 

Battlespace  Manager 

That  one  right  there.  Chaser  14,  that's  a  target. 

Commander 

Chaser  14. 

Battlespace  Manager 

And  that's  whole,  no  smoke. 

Commander 

Got  it,  Dave,  you  engage  that  with  a  PAM  and  engage 

Effects 

Which  one?  ' 

Commander 

Both  of  these  1  want  you  to  engage. 

Battlespace  Manager 

Right  there,  this  is  the  image  and  there  is  no  smoke  coming  and  that's  the 
chaser  picture. 

Effects 

Ok,  that's  the  one  you  shot  at  already,  right? 

Battlespace  Manager 

Yeah,  1  shot  it  with  a  cargo. 

Effects 

Ok,  so  shoot  the  chaser. 
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Battlespace  Manager 

So  shoot  that,  yeah,  Chaser  14. 

Commander 

Dave  do  you  want  someone  else  to  engage  that  Draega  7? 

Effects 

No,  I  got  it. 

Commander 

OK 

Blue 

Black6-Blue6 

Intel  Manager 

Ok,  the  radio  unknown... 

Commander 

Blue-Black 

Intel  Manager 

...good  pictures 

Blue 

I've  identified  the??  on  the  center  sector  unknown.  I  talked  to  your  2 
already. 

Blue 

It's  moving.  But  we  need  to  wait  to  develop  it.  Over. 

Commander 

Are  you  talking  about  along  the  PLOT?  Over. 

Blue 

But  the  problem  with  that  is  that  you  need  to  get  eyes  on  it,  but  once  you 
attack  it,  we  won't  be  able  to  do  that.  Over 

Commander 

Roger,  we're  holding  off  on  that  right  now. 

Effects 

How  about  the  MTLB,  what's  happening  over  there? 

Battlespace  Manager 

Sir,  I  made  that  unknown  into  an  MTLB. 

Effects 

OK,  so  I  got  rounds  in. 

Battlespace  Manager 

Another  radio. 

Effects 

I  got  him. 

Commander 

bk,  radio  north,  Dave,  once  you've  got  7... 

Effects 

I've  got  it. 

Commander 

Radio  link  15. 

Intel  Manager 

Radio  link  8  is  dead.  In  the  north,  the  first  one  in  the  north  that  one  that  was 
approaching  us. 

Commander 

Are  you  keeping  up  with  me  there? 

Effects 

Yeah  man,  I'm  there. 

Commander 

Got  it.  You're  on  it.  Thank  you. 

Effects 

LAM  going  in. 

Commander 

Good. 

Effects 

And  1  will... 

Intel  Manager 

bk  there  is  another  radio  unknown,  right  underneath  that  Dreaga  tank.  In  the, 
right 

Commander 

See  1  told  you  there  was  another  one  in  there. 

Commaner 

Airight,  the  PAM. 

Effects 

1  think  1  just  got  him. 

Battlespace  Manager 

1  think  Dave  Just  took  him. 

Commander 

Yeah  did  it  hit  though? 

Ah 

Yeah  it  hit,  but  which  vehicle? 

Techie 

Hey  Brooks,  can  you  turn  your  mike  on. 

Battlespace  Manager 

My  mike  is  on. 

Techie 

Alright. 

Battlespace  Manager 

Your  welcome. 

Intel  Manager 

Alright  it  was  fired  upon  once. 

Effects 

LAM  11  hit. 

Intel  Manager 

OK 

Commander 

Dave,  when  you  get  a  minute,  1  want  to  get  some  more  LAMs  up  north,  please. 

Battlespace  Manager 

Dave,  he  be  a  very  busy  man. 
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Commander 

Dave  is  a  busy  man  this  time,  and  that's  ok. 

Intel  Manager 

UAV  on  its  way. 

Commander 

Say,  Ken,  the  micros  are  not  helping  me  here. 

Intel  Manager 

Yes  they  are,  I've  got 

Commander 

In  the  north,  it  is,  but  1  want  to  go  back  at  take  a  look  at  that  PLOT  picture  of 
that  Dreaga. 

Intel  Manager 

It's  on  the  way  there. 

Commander 

Oh  is  that  the  one  in  the  center  going  back? 

Intel  Manager 

Going  back. 

Commander 

OK  good.  Where's  the  fourth  one? 

Intel  Manager 

There’s  3  in  the  north  there. 

Commander 

Where's  the  one  in  the  south? 

Intel  Manager 

All  4  are  in  the  north,  there's  one  in  the  center  heading  back. 

Commander 

Ah  1  see. 

Intel  Manager 

2  of  them  are  really  close. 

Commander 

1  got  it,  1  see  it  now. 

Intel  Manager 

1  got  2  pictures  of  that  dead 

Commander 

I'm  with  you.  You're  alright. 

Battlespace  Manager 

Ok.  All  my  snuffies  are  mounted.  We  are  preparing  to  move. 

Effects 

Better  run  like  hell. 

Battlespace  Manager 

We  shall.  We're  still  a  couple  of  minutes  away  from  moving  out. 

Intel  Manager 

1  can  see  my  recon  targets  moving. 

Commander 

Alrighty  now. 

Effects 

And  be  prepared  to  deal  with  the  scoutage. 

Battlespace  Manager 

Ok  a  PAM  on  chaser,  on  MTLB  was  a  hit. 

Effects 

Yeah 

Battlespace  Manager 

Vehicle  was  hit  at  radio  unknown  6.  Vehicle  was  hit  at  radio  unknown  13.  And 
vehicle  was  hit  at  radio  unknown  14.  So  we  are  getting  good  fires  in  depth,  but 
our  radios  are  not  transmitting  anymore. 

Commander 

I’m,  OK,... 

Commander 

Brooks,  before  we  cross  the  PLOT,  1  want  to  make  sure  we  cleaned  up  those 
Dreagas  in  the  center. 

Effects 

1  don't  know  how  to  do  that  for  you. 

Commander 

No,  no,  no,  micro  UAVs  headed  there. 

Intel  Manager 

Yeah,  he's  headed  right  there. 

Battlespace  Manager 

I’d  also  like  you  to  look  on  the  south  edge  of  the  north  wall.  In  that  outer  ridge 
of  the  north  wall. 

Intel  Manager 

At  the  edge  of  the  north  wall? 

Battlespace  Manager 

Right  now  I'm  talking  to  Jack. 

Intel  Manager 

Ok 

Battlespace  Manager 

You’ve  got  to  concentrate  on  what  Jack  told  you  to  do. 

Battlespace  Manager 

Jack, 

Commander 

Sir 

Battlespace  Manager 

...north  wall  up  here 

Blue 

Black  6,  Blue6,  we've  completed  the  SAR  paint  of  T1.  Over. 

Battlespace  Manager 

...based  on  what  we  had  as  far  as  the  truck.. .we  don't  have  anything  up  there.  , 

Commander 

1  guess  blue's  telling  me  he  completed  the  SAR  paint  for  T1 .  Where's  he  gotta 
go  next,  T2? 

Commander 

And  blue  black,  you  completed  T1  is  that  correct? 
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Blue 

That's  roger. 

Blue 

Got  about  70%  coverage  of  premium  targets  over. 

Intel  Manager 

If  he  could  cover  T2  that  would  be  the  next  step. 

Commander 

Roger,  can  you  move  to  T2,  please? 

Blue 

Standby 

Battlespace  Manager 

We  have  an  armor  mover  back  there  around  Brown  Debman.  Armor 

Commander 

1  wonder  if 

Effects 

Talk  to  me,  talk  to  me. 

Battlespace  Manager 

1  am  just  seeing  a  box  for  an  armor  indicator. 

Commander 

Is  that  MTLB  dead?  1  don't  see  a  lot  of  smoke  on  him. 

Effects 

All  1  know  is  that  1  hit  him. 

Battlespace  Manager 

Yeah,  I'm  reading  based  on  what  the  PAM  tells  us. 

Commander 

Yeah  but  I'm  looking  at  picture  of  him. 

Effects 

The  picture  there  is  pre-PAM. 

Battlespace  Manager 

Yeah  the  picture  was  from  the  chaser. 

Commander 

Oh,  1  got  it,  1  got  it. 

Battlespace  Manager 

Yeah  the  picture  was  from  the  chaser.  The  picture  was  from  the  chaser,  and 
the 

Commander 

I'm  with  you. 
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